An assessment for health education and health promotion in chronic disease demonstration districts: a comparative study from Hunan Province, China

Design for consultative indicators

By combining literature review with the related manuals of NCD demonstration districts drafted by the Chinese center for diseases control and prevention (CDC), we modified and designed a 38- consultative item for the Delphi process. These items were categorized into seven themes, including organizational management, financial support, professional personnel, health education and health promotion, heath awareness and behaviors of NCDs, management and control in NCD patients, others (such as satisfaction with supplies of health education and promotion, health literacy level) (Table S1).

Selection of experts for Delphi survey

We sent invitations to more than 20 potential experts via e-mail or letter, in which we explained to experts the study background, gave a brief introduction of Delphi method, and what experts needed to do. In total, 19 experts responded and agreed to participate in the Delphi process. Experts across the country were selected for this study using the following criteria:

• Be working for the department NCD control and prevention of CDC, health education institution, or public health research (especially on NCDs) in university.

• Have had work experience for at least five years at provincial or national institutions.

• Be interested in participating in this study.

Instrument for Delphi surveys

We listed the instrument for the Delphi surveys as the following four parts:

• General characteristics of the experts, including age, profession, professional title, education, years of working experiences and so on.

• A basic guideline for finishing the Delphi survey.

• Questionnaire with consultative indicators for evaluation of health education and promotion.

• Definition of experts’ authority both in familiarity with indicators and judgment criteria for the indicators.

Two-round Delphi process

With its anonymous consultation, feedback information, statistical inference, and easy utilization (Holbech et al., 2017; Kuster et al., 2015; Gracht, 2012), the Delphi technique is a popular assessment tool in health care research (Zhao et al., 2015; Balaguer et al., 2016 Jones & Hunter, 1995). We conducted a two-round Delphi process between late 2013 and 2014 as follows: during round 1, the experts needed to judge whether the items should be included based on validity and feasibility and were free to make comments. They also rated each item for importance on a 9-point Likert scale (Mrowietz et al., 2014; Suzuki et al., 2012). The scores from 1 to 9 correlate to “strongly unimportant” to “strongly important”, while 0 indicates disagreement.

The data from the round 1 were then summarized, revised, and resubmitted for round 2, following the same format as round 1 to determine quality indicators.

Consensus definition

The expert agreement, average scores, and coefficient of variation (CV) are internationally taken as common parameters for determining consensus in the Delphi process. However, there are controversies regarding the parameter thresholds (Hasson, Keeney & McKenna, 2010; Flores, Marshall & Cordina, 2014; Slade et al., 2014; Strosberg et al., 2015; Jang et al., 2015). We integrated the parameters with a modified criterion to avoid important items from potentially being removed.

In the first round, consensus level was set as: (1) Expert agreement (%): items were removed with agreement ≤40% and included when agreement ≥70%. They were considered uncertain when agreement ranged from 40% to 70%. (2) Median scores: items were excluded with scores ≤4, included when scores ≥7, and temporarily included when scores were between 4 and 6. (3) CV: items were included with CV <0.25. In the second round, consensus was reached only if agreement ≥70%, scores ≥7, and CV <0.25.

Selection of samples and RSR evaluation

After the Delphi process, ten districts from the Hunan province were selected and coded as samples for assessment with a simple randomizing function. This study conducted data collection from evaluated districts between 2014 and 2015 on schedule, when twenty-eight districts got the nomination in the province. Therefore, we randomly selected six districts or counties representing NCD demonstration districts. Meanwhile, we also randomly chose four districts or counties from the same cities as that of NCD demonstration districts, as control subjects from the rest of the non-demonstration districts in the province.

Finally, the RSR method was conducted in 2015 to assess the performance of the sample districts. As a comprehensive evaluation method, the RSR was developed by Sun & Tian, who also proved its validation and rationality (Sun & Tian, 1994; Wang et al., 2015b). The basic theory of the RSR method is that a dimensionless statistical indicator (RSR) is calculated from an n × m matrix through rank conversion (Wang et al., 2015b). With the indicator RSR, a parametric statistic by the linear regression equation ($R\hat{S}R$) was followed to analyze the RSR distribution. The statuses (worst/best) of evaluated districts were evaluated or classified based on the $R\hat{S}R$. The RSR procedure was carried out here through the following steps:

The original values of items were converted to high-quality values. There was no conversion work here due to their natural high-quality characteristics.

• Quality indicators within the ten sample districts were ranked based on their original values. All indicators were ranked in ascending order as R_ij (i ≤ n, j ≤ m)

• The RSR was calculated with the equation $RS{R}_i=\frac{{\sum }_{j=1}^m{R}_{ij}}{m\times n}$.

• The distribution of RSR was analyzed based on a Probit (-oriented downward accumulative frequency (p).

• A linear regression was set up. Taking RSR_i as a dependent variable and P as an independent variable separately, an equation was built using the formula as $R\hat{S}R=a+b\times$ Probit.

• With the $R\hat{S}R$, four levels (from best to worst) were classified into and set for the evaluation of the sample districts as follows: level I (<P_6.681, Probit <  3.5), level II (P_6.681, Probit: 3.5 ∼), level III (P₅₀, Probit: 5 ∼), and level IV (P_93.319, Probit: 6.5 ∼) (Sun & Xu, 2014).