Promoting data-driven decision-making in Jordan: strengthening national health information system and achieving consensus on core set of health system indicators

Abstract

Background

A well-functioning health information system (HIS) is foundational for strong health systems and the achievement of the Sustainable Development Goals. In Jordan, the national HIS, overseen by the Ministry of Health, faces challenges related to overlapping data collection, data availability gaps, and operational inefficiencies which compromise effective decision-making. This study aims to promote data-driven decision-making in Jordan by assessing the existing HIS and fostering consensus on a standardized set of indicators for core health system functions, maternal, child and adolescent health, and refugee health.

Methods

A multifaceted stepwise approach was adopted, encompassing the following steps: baseline assessment of HIS, compilation of a comprehensive list of candidate indicators, consensus meetings to prioritize and validate the indicators, and development of procedure manual for standardizing the shortlisted indicators.

Results

The baseline assessment of HIS identified areas for improvement at the following levels: governance and planning; infrastructure and resources; data management; and institutional capacity to support data-driven decision-making. Of 4,120 indicators reviewed from international sources and 215 from Jordan's indicators inventory, 415 candidate indicators were compiled and categorized into three priority thematic areas: core health and health systems indicators (n = 167), maternal, child and adolescent health indicators (n = 137), and refugee health indicators (n = 111). Fifteen stakeholders took part in the first consensus meeting, 14 in the second, and 10 in the third meeting. Utilizing a criterion-based ranking system, participants independently rated each candidate indicator against three criteria: Importance, Feasibility, and Actionability. The shortlisted indicators were subsequently validated against the criterion ‘retain’. This process resulted in a final validated list of indicators, comprising 55 core health systems indicators (33 of which are reported in Jordan); 40 maternal, child and adolescent health indicators (21 of which are reported in Jordan); and 26 refugee health indicators (none of which are reported in Jordan). Participants also suggested indicators to be added to each thematic areas. Three procedure manuals were developed and validated, corresponding to the three thematic areas.

Conclusion

Findings from this study can contribute to the broader discourse on HIS reforms in Jordan, emphasizing the need for ongoing efforts to enhance data quality, stakeholder collaboration, and infrastructure.

Résumé

Contexte

Un système d’information sanitaire (SIS) qui fonctionne bien est fondamental pour de solides systèmes de santé et la réalisation des objectifs de développement durable. En Jordanie, le SIS national, supervisé par le ministère de la Santé, est confronté à des défis liés au chevauchement des collectes de données, aux lacunes en matière de disponibilité des données et aux inefficacités opérationnelles qui compromettent l’efficacité des prises de décision. Cette étude vise à promouvoir la prise de décision basée sur les données probantes en Jordanie en évaluant le SIS existant et en favorisant un consensus sur un ensemble standardisé d’indicateurs pour les fonctions essentielles du système de santé, la santé maternelle, des enfants et des adolescent(e)s et la santé des réfugiés.

Méthodes

Une approche multidimensionnelle progressive a été adoptée, englobant les étapes suivantes : évaluation de base du SIS, compilation d’une liste exhaustive des indicateurs potentiels, réunions de consensus pour établir l’ordre de priorité et valider les indicateurs, et élaboration d’un manuel de procédures pour la normalisation des indicateurs présélectionnés.

Résultats

L’évaluation de base du SIS a identifié des domaines à améliorer aux niveaux suivants : gouvernance et planification ; infrastructure et ressources ; gestion des données ; et capacité institutionnelle à soutenir la prise de décision fondée sur les données. Sur les 4 120 indicateurs examinés à partir de sources internationales et 215 de l’inventaire des indicateurs de Jordanie, 415 indicateurs potentiels ont été compilés et classés en trois domaines thématiques prioritaires, soit les indicateurs de base de la santé et des systèmes de santé (n = 167), les indicateurs de santé maternelle, infantile et adolescente (n = 137) et les indicateurs de santé des réfugiés (n = 111). Quinze parties prenantes ont participé à la première réunion de consensus, 14 à la deuxième et 10 à la troisième réunion. À l’aide d’un système de classement basé sur des critères prédéfinis, les participants ont évalué indépendamment chaque indicateur potentiel en fonction de trois critères : Importance, Faisabilité et Capacité d’action. Les indicateurs présélectionnés ont ensuite été validés en fonction du critère « à conserver ». Ce processus a donné lieu à une liste définitive validée d’indicateurs, comprenant 55 indicateurs de base des systèmes de santé (dont 33 sont déclarés en Jordanie) ; 40 indicateurs de santé maternelle, infantile et adolescente (dont 21 sont déclarés en Jordanie) ; et 26 indicateurs de santé des réfugiés (dont aucun n’est déclaré en Jordanie). Les participants ont également suggéré d’ajouter des indicateurs à chaque domaine thématique. Trois manuels de procédures ont été élaborés et validés, correspondant aux trois domaines thématiques.

Conclusion

Les résultats de cette étude peuvent contribuer au débat plus large sur les réformes du SIS en Jordanie, en soulignant la nécessité de poursuivre les efforts pour améliorer la qualité des données, la collaboration des parties prenantes et l’infrastructure.

Plain language summary

In this study, we seek to strengthen Jordan's national health information system (HIS), which is overseen by the Ministry of Health, by assessing its current state and establishing a consensus on a standardized set of national indicators. A multistep approach was employed, involving a baseline assessment of the HIS, compilation of potential indicators, consensus meetings, and the creation of procedure manuals for the standardized indicators.Results from the baseline assessment highlighted areas for improvement in governance, planning, infrastructure, resources, data management, and institutional capacity of the HIS. A total of 415 candidate indicators were compiled from Jordan and international sources, focusing on core health systems, maternal, child and adolescent health, as well as refugee health. Stakeholders participated in consensus meetings, using a criteria-based ranking system to independently assess the indicators based on importance, feasibility, and actionability. The final set of validated indicators included 55 core health systems indicators, 40 maternal, child and adolescent health indicators, and 26 refugee health indicators. Participants also suggested additional indicators for inclusion.The study's findings offer valuable insights for national discussions among key stakeholders in Jordan, emphasizing the importance of continuous efforts to enhance data quality, stakeholder collaboration, and infrastructure for strengthening HIS. The approach, tools and manuals can be adapted by other countries to strengthen their own HIS, facilitating data-driven health policymaking on a global scale.

Background

Reliable and timely health information is the foundation of all health systems [1, 2]. It enables decision-makers at all levels of the health system to identify problems; make evidence-informed decisions on health policies and programs; and optimally allocate scarce resources, with the ultimate aim of achieving health improvements [3,4,5,6,7].

A well-functioning health information system (HIS) is critical for the attainment of the health-related Sustainable Development Goals (SDGs) [8], particularly 'SDG 3: Good Health and Well-being’, as it supports targeted health interventions and robust monitoring of health outcomes, notably in areas such as maternal, child, and adolescent health. Additionally, HIS plays a key role in advancing 'SDG 5: Gender Equality’ and ‘SDG 10: Reduced Inequalities’ by identifying disparities and facilitating efforts and equitable resource allocation to improve health outcomes across diverse populations, thereby enhancing health equity [9, 10]. A robust HIS enables the planning and implementation of actions that mitigate health inequities at all care levels, aligning with the SDG mandate to ‘Leave No One Behind' and ensuring equitable access to essential services like reproductive health [11]. Increasingly, the emphasis on the accuracy and reliability of health data collection has been recognized as pivotal in enhancing health systems and achieving these goals [12].

The COVID-19 pandemic further re-enforced the indispensable role of HIS and the need for leveraging the capabilities of existing HIS to collect, transmit and analyze health data in real-time to understand the epidemiological situation and craft appropriate data-driven strategies and responses [13,14,15]. During such crises, HIS offers the necessary evidence to make informed decisions, helping policymakers and health professionals act wisely and change their strategies when needed to improve health outcomes [10].

While there is a growing realization of the need for effective and sustained action to strengthen HIS, many countries in the Middle East and North Africa (MENA) region lack well-functioning HIS to support health system strengthening [4, 16,17,18,19]. Key challenges include: 1) production of poor-quality data (inaccessible, inaccurate, incomplete, or irrelevant data) that does not align with decision-makers’ needs; 2) duplication, fragmentation and unnecessary production of data that become a burden and a barrier to effective information use; 3) a critical shortage of skilled health personnel who are equipped with the knowledge and capacity to collect and report essential health indicators and 4) poor dissemination of data in decision-making and health systems strengthening. Without proper actions aimed at strengthening the existing HIS and improving the use of data produced by HIS, health systems may never fully be able to meet the needs of the populations they serve, and countries will fail to achieve the SDGs by 2030 [20].

This project seeks to strengthen the HIS in two low- and middle-income countries (LMICs) in the MENA region—specifically, Lebanon and Jordan. In this paper, we present the findings from the research conducted in Jordan.

Jordanian context

Jordan is an upper middle-level income country with a total area of 89 342 square kilometers divided into 12 governorates. The population size reached 10.5 million in 2019 [21]. The country has a constitutional monarchy with a Prime Minister as head of Government appointed by His Majesty the King [21]. The Higher Health Council is the authorized entity to develop health policies and prepare national strategies for the health system.

Jordan’s health care system is a complex amalgam of three major sectors: public, private, and donors. It is relatively fragmented (between the public and private sectors and also within the public sector) and hospital-centric [21]. The main health care provider is the public sector, consisting of three providers: Ministry of Health (MOH), the Military Royal Medical Services (RMSs) and two university-affiliated hospitals (UHs) as well as the Centre for Diabetes, Endocrinology and Genetics. These organizations finance and deliver healthcare services to approximately 70% of the population, including civil service employees and members of the military as well as their dependents. The private sector includes private hospitals and diagnostic and therapeutic centers in addition to hundreds of private clinics. The international sector and charitable sectors provide services through UNRWA clinics for Palestinian refugees and the UNHCR and King Hussein Cancer Center and charity association clinic. [22,23,24].

The Health Information Centre, established in the MoH in 1992, is overseen by the Minister of Health. The directorate of Health Information, Studies and Research established within the MoH (later linked as ‘Department of Information Studies’ to the Institutional Development and Quality Control Directorate in 2019) collects basic health information from MoH facilities and other public and private hospitals. The Directorate produces the MoH annual report which is considered the main source of information about health services, including health human resources. Jordan’s HIS obtains data from different type of sources such as various institutions and population-based data sources, including [25]:

• MoH health centers and hospitals (all healthcare services).

• Royal Medical Services (RMS), university

• Hospitals, and private Hospitals.

• Department of Statistics information on health population indicators.

• Ministry of Health Directorates (communicable, non-communicable, occupational, school, environment, chest and migrant and organ plant directorates) through the annual periodic reports from central directorates in the Ministry of Health.

• Civil registration and vital statistics (births and deaths).

• Integrated Electronic Reporting System

• Jordan Cancer Registry Report (bi-annual reporting).

• Annual forensic report.

• Different activities of central directorates at the Ministry of Health (e.g. Licensing directorate).

• Financial data from the MoH (including supplies and medications).

• Annual Health Insurance financial reports.

• National Health accounts available from the Higher Health Council

While significant strides have been made in developing the HIS over the past decade, including the launching of Hakeem, a national e-health program, with the goal of revolutionizing Jordan's healthcare system [26], the need to strengthen the HIS has been reiterated in the National Strategy for Health Sector in Jordan [23]. Priorities identified by the MoH relate to reinforcing the HIS, including civil registration, risk factor and morbidity monitoring and health systems performance; and promoting evidence-based health policy development and effective utilization of knowledge management systems [27].

Existing studies within Jordan indicate that the HIS struggles with fragmentation and effective coordination and integration across different data sources. It has been noted that data from various sources are not well coordinated within the HIS framework, leading to overlapping data collection, gaps in available data, challenges in readiness, and resource wastage [28]. Additionally, the HIS lacks effective linkages and integration with routine information from various departments within the MoH and other ministries, such as the Civil Registration and Vital Statistics [28]. This complexity is compounded by outdated population survey tools, incomplete survey modules, and underutilized institutional data capabilities, which hinder efficient data utilization and impact the formulation of data-driven national health policies [28].

Within the realm of maternal, child and adolescent health, there is still a pressing need to integrate family planning services into primary health care, adopt preconception care within the public maternal and reproductive health services, and strengthen the maternal mortality surveillance and response system [8]. Additionally, Jordan faces challenges in adolescent health due to the lack of standardized data and indicators, as the HIS is not well-equipped to capture this information [27].

Furthermore, Jordan is among the countries most severely impacted by the Syrian refugee crisis, hosting the world's second-highest proportion of refugees relative to its population [29, 30]. However, the country's HIS is still not well-designed to support disaggregation by refugee status, thus making it difficult to ascertain coverage rates of key health indicators for refugees [31]. This need for data to be disaggregated by refugee status represents an important area for improvement of the country’s HIS.

A recent assessment highlighted that while Jordan’s HIS does support tracking of progress towards social policies and actions, it falls short in effectively addressing health inequities [32]. The system includes measures of some social determinants of health but lacks a comprehensive database and necessary integration to effectively measure and address health disparities [32]. Additional challenges faced by Jordan’s HIS include constrained financial resources, a lack of regulatory and policy support specifically tailored to health information systems, and inadequate standardization in the methods of data collection, processing, and analysis, which limits the utility of the data for research purposes [33, 34]. There is also a notable deficiency in skilled IT personnel capable of managing and advancing HIS capabilities [33, 34].

In light of these issues, there is a need to strengthen the existing HIS and promote the collection of relevant, reliable and comprehensive data and indicators to allow for improved health planning and decision-making in Jordan.

Aim and objectives

The overall aim of this study is to promote data-informed decision-making in Jordan, by strengthening the existing HIS and ensuring the right sets of indicators are in place for generating relevant, reliable and timely data to effectively inform decisions and support health systems strengthening. In line with the SDG’s commitment to “leaving no one behind” and the articulation of the importance of addressing vulnerabilities on a global scale, specific efforts will be invested to optimize indicators related to maternal, child and adolescent health as well as refugee health-related indicators.

This study seeks to answer the following question: How can the current HIS in Jordan be strengthened to capture quality, relevant and standardized data with a focus on vulnerable populations?

The specific objectives are to:

• Conduct a baseline assessment of the existing national HIS structures and capacities, including those of staffs, directors and heads of departments within the MoH directorates

• Achieve consensus on a core set of health system indicators that are relevant and useful for health decision-making as well as optimize indicators for vulnerable populations including mothers, children, adolescents and refugees

• Standardize data collection process for the core set of selected HIS indicators

To the best of our knowledge, this is the first experience from the MENA region to report on the selection, development and validation of standardized health system indicators for informing health decision-making and strengthening health systems.

Methods

A multifaceted stepwise approach was adopted, encompassing the following steps:

• a. Baseline assessment of the health information system

• b. Compilation of a comprehensive list of candidate indicators

• c. Consensus meetings to prioritize and validate indicators

• d. Development of procedure manual for standardizing the shortlisted indicators

A local liaison team was established in Jordan to coordinate stakeholder engagement and co-lead implementation of the project in the country. Data collection spanned from 2019 until 2022.

Data collection

• a. Assessment of health information system

A tools manual was developed based on the Performance of Routine Information System Management conceptual framework. This framework adopts a multifaceted approach to evaluate routine health information system performance by examining the causal pathways between the organizational, technical and behavioral determinants of performance. The tools manual included the following sub-tools:

• Sub-tool 1: RHIS Mapping Tool

• Sub-tool 2: RHIS Profile Overview

• Sub-tool 3: RHIS Performance Diagnostic Tool

• Sub-tool 4: RHIS Organizational and Behavioral Assessment Tool

• Sub-tool 5: RHIS Management Assessment Tool

The tools manual was pilot-tested in collaboration with the local liaison team as part of a one-day workshop in December 2019. Following the pilot-testing, the local liaison team coordinated the administration of the tools manual with key selected stakeholders in Jordan. Data collection for the different sections of the tools manual involved a mix of interviews, focused group discussions and self-administered surveys. Where applicable, participants were asked to provide documentations to supplement and cross-reference their responses. Participants were selected based on their relevance to the different sections of the tools manual. An overview of the target audience and process of administration of the different sub-tools within the tools manual is presented in Table 1.

Table 1 Target audience and process of administration of the different sub-tools within the tools manual

• b Compilation of a comprehensive list of candidate indicators (i.e. initial selection of indicators)

This step drew on a review of international health system indicators, and an inventory of health indicators reported in Jordan.

We conducted an extensive literature review of international health and health system indicators to select candidate dimensions and indicators for inclusion in the national HIS of Jordan. We ran broad and specific searches corresponding to core health systems; maternal, child and adolescent health; and refugee health-related indicators. We searched both published and grey literature. For the published literature, we searched PubMed and Google Scholar using the following keywords: (health system OR maternal OR child OR adolescent OR refugee) AND (indicator* OR data OR reporting system). For the grey literature, we searched intergovernmental websites and reports. A brief overview of the different sources is provided below, by thematic areas:

• Core health and health systems indicators: A comprehensive list of health and health systems indicators was compiled by reviewing existing indicator sets from the published literature and leading organizations such as the World Health Organization (WHO), Centers for Disease Prevention and Control (CDC), United Nations Statistics Division, United Nations Children's Fund (UNICEF), Pan American Health Organization (PAHO), Organization for Economic Co-operation and Development (OECD), and World Bank [27, 35,36,37,38,39,40,41,42,43,44,45]. The following two sources were used as starting points for indicator compilation: 1) the WHO Global Reference List of 100 Core Health Indicators (plus health-related SDGs) which includes all SDG 3 indicators [40] and; 2) the core indicators for monitoring health and health system performance in the Eastern Mediterranean Region [27]. These were then supplemented by additional indicators identified from the other organizations and the published literature.

• Maternal, child and adolescent health indicators: We adapted the Health Equity Assessment Toolkit (HEAT) developed by the WHO which contains the WHO Health Equity Monitor database. This database allows users to assess in-country inequalities against over 30 Reproductive, Maternal, Newborn and Child Health (RMNCH) indicators and six dimensions of inequality (economic status, education, place of residence, subnational region as well as age and sex (where applicable) [40]. This was supplemented by indicator lists from UNICEF [46], and two scoping reviews on maternal and child health [47, 48]. As for adolescent health, we utilized the list of 32 indicators generated during the WHO Technical consultation meeting on indicators of adolescent health as a starting point for prioritization efforts. These indicators have been chosen on the basis of four criteria: public health relevance, validity of constructs, feasibility of measurement and possibility of detecting change over time [49]. The indicators were complemented by additional indicators proposed by other sources.

• Refugee health-related indicators: Data from United Nations High Commissioner for Refugees (UNHCR) [50] health information system was supplemented by additional indicators from UNRWA and the published literature on refugee health.

For the Jordanian indicator inventory, we conducted a comprehensive mapping exercise to pinpoint the health system indicators presently captured by the national HIS in Jordan. Data sources included official reports from the MOH and Department of Statistics, annual statistical reports such as Population and Family Health Surveys, and national health accounts. Following this, the compiled indicator inventory underwent scrutiny and validation by representatives from the MoH.

We cross-checked, refined and consolidated the indicator lists derived from the international document review with the indicator inventory specific to Jordan. The consolidated candidate indicators were subsequently categorized under three priority thematic areas, namely: Core health and health system indicators; maternal, child, and adolescent health indicators; and refugee health indicators. Within each thematic area, we applied the WHO conceptual framework [27] to further stratify the indicators into the following domains: health status, risk factors, service coverage, and health system performance.

• c Consensus meetings to prioritize and validate the indicators

Three consensus meetings were held with policymakers and stakeholders in Jordan between July 2019 and February 2022 to prioritize and shortlist the initially consolidated indicators and subsequently validate the shortlisted indicators within each thematic area (core health and health systems; maternal, child and adolescent health; and refugee health). In the first meeting, participants prioritized and shortlisted the “core health and health systems” and the “maternal, child and adolescent health” indicators. In the second meeting, participants validated the shortlisted indicators from the first meeting. In the third meeting, participants prioritized and shortlisted the “refugee health” indicators followed by subsequent validation of the shortlisted indicators.

A two-round modified Delphi technique was employed to derive a final set of indicators (within each thematic area) for inclusion in the national HIS for Jordan. For the prioritization and shortlisting of candidate indicators, participants individually ranked each candidate indicator against the following 3 criteria: Important; Feasible and Actionable (see Table 2). These criteria were adapted from a study by Al-Katheeri el at 2018 [51] and revised based on stakeholder consultations in Jordan. For the validation of the shortlisted indicators, participants reviewed each indicator and indicated whether it should be retained. For an indicator to be retained, it should meet a list of criteria, as depicted in Table 2. Following the validation of indicators, participants were asked to select a subset of those indicators for visualization and regular reporting (see Table 2). For both the prioritization and validation phases, participants had the opportunity to list any additional indicators or suggest modifications to the wordings of the existing indicators.

Table 2 Criteria used for prioritization and validation of indicators

Each consensus meeting involved a diverse group of 12–15 individuals purposively selected based on the below sampling framework:

Senior and middle-level policymakers from the public sector such as officials from the MoH and related ministries

• Representatives from High Health Council in Jordan

• Representatives from professional associations

• Healthcare directors, managers and professionals

• Representatives of NGOs, UN agencies and other non-state sector who are involved in refugee work or collect data related to refugee

• Researchers and academics who are active in the realm of health systems research

• Other stakeholders who use data from the national health information system to inform their decisions

The sampling also took into consideration gender balance.

The meetings were informed and guided by three prioritization and validation tools, corresponding to the core health and health systems indicators; maternal, child and adolescent indicators; and refugee health indicators, respectively (see Supplementary file 1). Each tool was filled on an individual basis by participants.

• d Development of procedure manuals

Procedure manuals enable the standardization of all aspects of data handling from collection, quality-assurance and flow, to processing, compilation and analysis. This can provide guidance on key aspects including individuals responsible for data collection, type and methods of data collection, timing of data reporting, and data interpretation and use. When utilized in a systematic and coordinated fashion by all health partners, these resources help to ensure that data and indicators are comparable across settings and achieve compatible degrees of aggregation [52].

Procedure manuals were developed for the short-listed indicators. The development of the content for the procedure manuals was guided by the WHO global metadata registry and the Standards and Best Practices for Data Sources [3, 53]. The procedure manuals were validated through pre-testing to enable assessment of the technical feasibility of data gathering. Specifically, a small working group encompassing representatives from the MoH hospitals and primary care was established to validate the feasibility and applicability of the procedure manuals within the Jordanian context, including the available and preferred data sources for each indicator.

Data analysis

For the baseline assessment of HIS structures and capacities, data generated from the different sources were collated and analyzed in aggregate form and categorized according to the components of the tools manual. Data triangulation helped provide a more in-depth understanding of the issue and increase the reliability and validity of findings through cross-checking of information across different data sources. For the development of action plan for improvement, we made use of both thematic and framework analysis. Specifically, we adapted the framework by MEASURE Evaluation for rapid assessment of routine HIS to categorize the emerging themes into four strategic areas for improvement: Governance and planning; Infrastructure and resources; Data management;and Institutional capacity building [54]. In addition to coding the findings under predefined themes within each strategic area, we also allowed for new themes to emerge inductively.

For the indicators, separate scores were generated for each thematic area: core health system, maternal, child and adolescent health, and refugee health. Shortlisting of indicators was done against three criteria: Important, Feasible and Applicable. Participants rated each indicator on a binary scale (Yes/No) for each criterion. Average scores were calculated for each criterion per domain. Indicators that scored above average for all three criteria were shortlisted for validation. The short-listed indicators were validated based on the criteria ‘retain’. Indicators scoring above average for retention criterion were included in the final list for the HIS. Participants were also asked to identify those shortlisted indicators with potential for visualization.

Results

Assessment of health information system

Findings from the baseline assessments revealed lack of well-functioning national HIS that can support health system strengthening. Data sets were often fragmented, patchy, and compartmentalized. Institutional structures, systems and culture were not conducive for MoH to operate as proactive knowledge management organizations. Existing personnel lacked the capacity to manage and advance HIS capabilities and promote data-driven decision-making.

Actions for improvements encompass four strategic areas which interact with each other to strengthen the HIS and facilitate better-informed decisions (see Table 3 and Supplementary file 2 for the detailed action plan):

• Governance and planning

• Infrastructure and resources

• Data management

• Institutional capacity building

Table 3 Strategic areas for improvement

Compilation of a comprehensive list of candidate indicators

As indicated earlier, this step drew on a review of international health system indicators, and an inventory of health indicators reported in Jordan.

The review of international sources retrieved an initial list of 4,120 indicators (Fig. 1). A first round of assessment by the research team refined the list by removing duplicates and eliminating indicators that are not related to health systems (e.g., clinical indicators) or are not aligned with the national scope (e.g., facility/organizational-specific indicators). Furthermore, we excluded input indicators that focus on the existence of policies and strategies (e.g., policies on breastfeeding) and merged similar indicators into one. This process resulted in 545 potentially relevant indicators from international sources (Fig. 1).

Fig. 1

Flowchart detailing the process of indicator selection

Additionally, 215 indicators were initially compiled from the different national sources within Jordan. Supplementary file 3 provides an overview and evolution of indicators from Jordan’s national HIS. The research team collaborated with the local liaison team in Jordan to review the list, removing duplicates (n = 4), filtering out indicators that are specific to a single health facility or organization (n = 28), and merging similar indicators into one (n = 79). This process resulted in a refined list of 104 indicators already reported in Jordan.

The refined lists of indicators from Jordan (n = 104) and international sources (n = 545) were combined to create the initial draft of the Priority Setting Tool. This draft tool contained 649 indicators. In a second round of assessment, we collaborated with the local liaison team in Jordan and an expert from the MoH to further remove duplicates, merge similar indicators into one, discard those without valid measurement methods, and categorize detailed indicators into broader categories. This resulted in the selection of 415 candidate indicators for subsequent prioritization and validation by key policymakers and stakeholders in Jordan. Of these, 84 were already being reported within Jordan’s HIS. In other words, 80.7% of the identified indicators for Jordan were integrated into the tool. The remaining 331 indicators comprising the tool were newly identified from international sources.

The 415 indicators were subsequently categorized under three thematic areas (corresponding to three priority setting tools), namely: core health and health systems indicators (n = 167); maternal, child and adolescent health indicators (n = 137); and refugee health-related indicators (n = 111). Within each thematic area, the indicators were further stratified into the following four domains: health status, risk factor indicators, service coverage indicators and health system indicators.

Prioritization and validation of the indicators

Fifteen participants took part in the first, 14 in the second, and 10 participants in the third consensus meeting. Each meeting included a diverse mix of participants from the High Health Council, the Jordanian Ministry of Health, Jordanian Royal Medical Services (RMS), Syndicate of Physicians, Syndicate of Nurses, Private Hospitals Association, and the Association of Public Health, WHO-Jordan and UNHCR Jordan.

In the initial round of prioritization, participants independently assessed the indicators under each thematic area. Utilizing a criterion-based ranking system, they rated each indicator against 3 criteria: Importance, Feasibility, and Actionability. This led to the preliminary selection of 55 core health systems indicators; 57 maternal, child and adolescent health indicators; and 33 refugee health indicators. In the second round, these shortlisted indicators were validated against the criterion ‘retain’. This resulted in a final validated set of indicators, comprising of 55 core health and health systems indicators, of which 33 (60%) were already reported in Jordan; 40 maternal, child and adolescent health indicators, with 21 (52.5%) already reported in Jordan; and 26 refugee health indicators, none of which were being reported in Jordan. Participants also suggested indicators to be added to each of the three thematic areas.

Of the indicators initially being reported in Jordan and included in the prioritization tools, 35 (41.6%) were subsequently retained as part of this exercise (Supplementary file 3).

The detailed findings are presented below, by thematic area.

• i Core health and health systems indicators

The initial list comprised 167 core health system indicators. At the time of data collection, 58 of these indicators, representing 33% of the total, were being reported within Jordan's national HIS. Respondents rated each indicator against three criteria: importance, feasibility, and applicability. Fifty-five indicators scored above the average for all criteria and were subsequently shortlisted for validation. Table 4 summarizes the average scores per criterion and per domain.

Table 4 Summary of average scores per criteria and domain (prioritization)

The 55 shortlisted indicators were subsequently validated based on average agreement per domain on the ‘retention’ criterion (Table 5). This resulted in a final validated set of 55 core health systems indicators (10 in dimension 1, 9 in dimension 2, 3 in dimension 3, and 32 in dimension 4). Among these 55 retained indicators, 33 (60%) were already being reported within Jordan's national HIS. Moreover, 17 indicators were identified as suitable for visualization. Participants also suggested additional indicators for inclusion.

Table 5 Summary of average scores per criteria and domain (validation)

Table 6 provides an overview of the final shortlisted and validated set of core health and health system indicators, including those already being reported within Jordan's national HIS at the time of data collection and those identified as suitable for visualization. Participants individually suggested 17 additional indicators for inclusion in the final set. These indicators were subsequently refined by the research team, based on the literature (Table 7).

Table 6 Final shortlisted and validated set of core health and health systems indicators

Table 7 Additional suggested core health systems indicators and their refinement

• ii Maternal, child and adolescent health indicators

The initial list consisted of 137 indicators (100 maternal and child health, and 37 adolescent health indicators). At the time of data collection, 41 of these indicators (36 for maternal and child health, and 5 for adolescent health), collectively representing 30% of the total indicators considered for prioritization, were being reported within Jordan's national HIS.

Participants rated each indicator against three criteria: importance, feasibility, and applicability. Average scores were calculated for each criterion, stratified by indicator type (maternal and child vs. adolescent health) (see Table 7). Forty-one maternal and child health indicators (dimension 1 (n = 13), dimension 2 (n = 18), dimension 3 (n = 18) and dimension 4 (n = 9)) and 16 adolescent health indicators (dimension 1 (n = 11), dimension 2 (n = 3), dimension 3 (n = 2), and dimension 4 (n = 10)) scored above average for all of the criteria and were subsequently shortlisted for validation. Table 8 summarizes the results.

Table 8 Summary of average scores per criterion and domain (prioritization)

The shortlisted indicators were subsequently validated based on average agreement per domain on the ‘retention’ criterion (Table 9). The final set included 40 indicators: 32 maternal and child health indicators and 8 adolescent health indicators. Among these 40 retained indicators, 21 (52.5%) were being reported by Jordan's national HIS (19 for maternal and child health, and 2 for adolescent health) at the time of data collection. Moreover, 25 indicators were identified as having potential for visualization. Participants also suggested additional indicators for inclusion in the final set.

Table 9 Summary of average scores per criteria (validation)

Table 10 provides an overview of the final set of shortlisted and validated maternal, child and adolescent health indicators, including those already being reported within Jordan's national HIS at the time of data collection and those identified as suitable for visualization. Participants also individually suggested 19 additional indicators for inclusion in the final list. These indicators were subsequently refined by the research team, based on the literature (Table 11).

Table 10 Final shortlisted and validated set of maternal, child and adolescent health indicators

Table 11 Additional suggested maternal, child and adolescent health indicators and their refinement

• iii Refugee health indicators

The initial list contained 111 refugee health indicators. It is important to note that, at the time of data collection, none of these indicators were reported by the national HIS. However, 60 indicators were tracked by International Non-Governmental Organizations (INGOs) including entities like the United Nations High Commissioner for Refugees (UNHCR). Respondents rated these indicators against three criteria: importance, feasibility, and applicability. Thirty-three indicators scored above average for all of the criteria and were subsequently shortlisted for validation (Table 12).

Table 12 Summary of average scores per criteria (prioritization)

The 33 shortlisted indicators were subsequently validated based on average agreement per domain on the ‘retention’ criterion (Table 13). The final set included 26 refugee health indicators (domain 1 (n = 8), domain 2 (n = 8), domain 3 (n = 8), domain 4 (n = 2)). While none of these core indicators were reported by Jordan's national HIS, 16 were tracked by INGOs or other external agencies at the time of data collection. Additionally, 14 indicators were identified as having potential for visualization. Participants also suggested additional indicators for inclusion in the final set.

Table 13 Summary of average scores per criteria (validation)

Table 14 provides an overview of the final shortlisted and validated set of refugee health indicators, including those suitable for visualization and those currently reported by Jordan’s national HIS. Participants also individually suggested 4 additional indicators for inclusion in the final list (Table 15).

Table 14 Final shortlisted and validated set of refugee health indicators

Table 15 Additional suggested indicators for refugee health

Development of procedure manuals

Three procedure manuals were developed, corresponding to the three prioritized thematic areas: core health system indicators; maternal child and adolescent indicators; and refugee health indicators, respectively.

Each procedure manual provided the following information for each included indicator (see Fig. 2):

• Indicator

• Alternative indicator

• Definition

• Rationale

• Type of indicator

• Unit of measure

• Formula

• Target

• Frequency of collection

• Frequency of dissemination

• Denominator inclusion criteria

• Denominator exclusion criteria

• Numerator inclusion criteria

• Numerator exclusion criteria

• Preferred data sources

• Other data sources

• Limitations/comments

• References

Fig. 2

Example of data collection sheet for the indicator ‘mortality’

The procedure manuals were validated through pre-testing to enable assessment of the technical feasibility of data gathering. The revised manuals were subsequently shared with key policymakers and stakeholders in Jordan. Supplementary file 4 provides a sample procedure manual for maternal, child and adolescent health.

Discussion

To our knowledge, this is the first study from the MENA region to undertake a comprehensive assessment of the national HIS and achieve consensus on a core set of health and health systems indicators including for maternal, child adolescent and refugee health. The COVID-19 pandemic reaffirmed the criticality of a robust HIS with valid and reliable indicators for more effective and timely decision-making [14, 55]. Findings from this study can help foster a culture of sharing and using data by ensuring the right indicators, capacities and resources are in place to promote well-functioning HIS that can inform policymaking and practice.

The baseline assessments of the national HIS structure and capacities revealed gaps and highlighted areas for improvement at the following levels: governance and planning; infrastructure and resources; data management and institutional capacity. Strengthening these areas is crucial to support data-driven decision-making, which is vital for advancing public health outcomes and enhancing the efficiency of health systems in Jordan.

Importantly, through this study, we were able to achieve consensus on a standardized set of national health system indicators, which has significant implications for healthcare planning and resource allocation in Jordan. Of the 4,120 indicators reviewed from international sources and 215 from the indicators inventory in Jordan, 415 candidate indicators were compiled and categorized into three priority thematic areas: core health system indicators (n = 167), maternal, child and adolescent health indicators (n = 137), and refugee health indicators (n = 111). To ensure the selection of appropriate indicators and also increase the likelihood that indicators will have wider acceptance, we involved various stakeholders, including policymakers, health professionals, health researchers and representatives of non-governmental and intergovernmental organizations, in prioritizing, shortlisting and validating the indicators. This resulted in a final shortlisted and validated set of 121 indicators, comprising 55 core health systems indicators, of which 60% were already reported in Jordan; 40 maternal, child, and adolescent health indicators, with 52.5% reported in Jordan; and 26 refugee health indicators, none of which were reported in Jordan. A striking finding is that less than half (41.6%) of the indicators already reported in Jordan's national HIS and included in the priority setting tool were ultimately retained after the prioritization and validation exercises. This underscores the urgent need for revising and integrating new indicators into the national HIS to ensure it can provide a comprehensive and effective framework for ongoing health monitoring and data-driven decision-making. Participants also suggested indicators to be added to each of the three thematic areas. While several of these suggested indicators were included in the initial priority setting tools, they were not retained in the final selection. Consequently, these indicators are listed separately for future consideration, indicating potential areas for expanding the HIS to better meet the needs of the health system.

Another notable finding is the absence of refugee health indicators in the national HIS of Jordan, with most relevant data being managed by external agencies and INGOs such as UNHCR. The national system's current data stratification—categorized by nationality for some indicators (Jordanians and non-Jordanians)—provides limited insights specific to the substantial Syrian refugee population in Jordan, highlighting a gap in equity-related health data. As of January 2024, around 730,000 Syrian refugees are registered with the UNHCR in Jordan [56], with an estimated 80% living below the poverty line and 60% in extreme poverty [57]. The ongoing collection of data for key demographic groups such as mothers, children, adolescents, and refugees is essential. It allows for the monitoring of individual risk factors and the identification of both positive and negative health trends [58]. These data are invaluable for prompt health policy decisions and can significantly influence the overall health status of the population [58].

The indicators generated from this study comprise an initial core set of indicators that can be further revised and incrementally scaled up once the system is well-established and sufficiently mature to collect, measure, report and respond to the needs of policymakers and decision-makers. Policymakers can use the identified indicators to prioritize areas requiring intervention, allocate resources efficiently, and monitor the impact of health programs over time. Standardized data collection protocols and improved reporting mechanisms are also crucial for generating actionable insights and promoting informed decision-making at both the policy and healthcare delivery levels. As such, the procedures manuals generated from this study were subsequently shared with policymakers and practitioners to carry forward the task of liaising with different stakeholders to identify data sources, gaps, data analysis needs, and decision support systems that need to be scaled up to support this undertaking.

Effective governance of indicator development, monitoring, and periodic updates is essential to maintain a high-functioning health system. It is advisable to involve a diverse array of stakeholders, including policymakers, healthcare providers, patient representatives, and academic researchers in these processes to promote inclusive and more equitable decision-making [59]. Governance should involve clear protocols for regularly assessing the relevance and effectiveness of indicators, reflecting evolving health challenges and outcomes. By institutionalizing roles such as a steering committee, project management team, data quality advisory group, and expert panel within government structures, stakeholders can continuously evaluate and update national indicator sets as necessary. It is also vital to adopt the learning health systems approach which relies on a bidirectional flow of information between data sources and users to ultimately improve outcomes and enhance efficiency [60]. Incorporating feedback from end-users can glean insights into their experiences with the indicators and gather their recommendations on draft criteria for evaluating them [59]. This in turn, ensures that indicators remain relevant and aligned with the dynamic and evolving health needs and priorities of all community segments [61].

Moreover, the implementation cost considerations are critical. Detailed budgeting and securing sustainable funding are imperative to support the ongoing operations and expansion of health information systems. Anticipated costs include training personnel, technology updates, data management, and security measures [62]. These technical, human, social and organizational considerations should be accounted for in any national level plan for enhancing a HIS. This is particularly in low and middle-income countries and those in the early stages of establishing health information systems which often face challenges in obtaining the necessary resources for development, implementation, and maintenance of HIS [55].

Lastly, once the HIS is sufficiently mature and the capacity is built for collecting and reporting evidence-based performance indicators, considerations could be given to prioritize the selection of a subset of indicators that facilitate meaningful within-country and cross-country comparisons, thereby enhancing the relevance and utility of findings for decision-makers at both national and regional levels. Selecting such indicators not only enables the identification of variations and disparities within different regions of Jordan, but also the benchmarking of performance against regional peers in the MENA region. This comparative analysis is vital for policymakers and stakeholders to assess the strengths and weaknesses of the Jordanian health system, identify best practices, and implement targeted interventions to improve health outcomes and healthcare delivery. It also promotes regional benchmarks and fosters a competitive drive towards healthcare excellence, which is instrumental in driving systemic improvements [63].

Some of the challenges and potential limitations of this study are worth noting. Firstly, the data collection period coincided with the onset of the COVID-19 pandemic, resulting in numerous delays and the rescheduling of initially planned stakeholder meetings. This situation also impacted our ability to convene the same group of stakeholders for both prioritization and subsequent validation sessions. Despite these challenges, we endeavored to mitigate the disruptions by arranging targeted follow-up meetings and seamlessly transitioning to online interviews and group discussions when necessary. Additionally, to accommodate participants, tools and manuals were distributed via email for input in certain instances. Moreover, we engaged a diverse array of stakeholders, including policymakers, healthcare professionals, researchers, and representatives from non-governmental and intergovernmental organizations, to ensure the selection of relevant indicators and foster broader acceptance of the chosen metrics. Secondly, despite our attempt to develop and validate a core set of national indicators for regular reporting, the additional indicators suggested by participants warrant further review and validation with a diverse group of stakeholders in Jordan. Finally, while this study describes the process of selecting, developing, and validating core set of indicators, it does not discuss the actual implementation of indicators or the process for using results of indicators in policymaking or regulation. In future phases, we aim to examine the implementation and use of the indicators in healthcare quality improvement and policymaking in Jordan.

Conclusion

In this study, we conducted a baseline assessment of the national HIS in Jordan and achieved consensus on a standardized set of national indicators for the core health system, maternal, child and adolescent health, and refugee health, respectively. The findings and gaps emerging from this study can act as a basis to inform national deliberations and dialogues among key stakeholders in Jordan. They can also contribute to the broader discourse on HIS, emphasizing the need for ongoing efforts to enhance data quality, stakeholder collaboration, and technological infrastructure. Furthermore, the study's approach, tools, and manuals can be adapted by other countries looking to strengthen their HIS and promote data-driven health policymaking and practice.