Reviews on common objectives and evaluation indicators for risk communication activities from 2011 to 2017

Inclusion and exclusion criteria

The inclusion criteria used to select empirical studies were (1) that the study evaluates risk communication activities in any field, and (2) that the study is written in English. This study also included previous research that did not directly evaluate risk communication but did ask implementers, such as medical professionals, about the objectives and effects of their activities relevant to risk communication. These studies were included because they provided insight into the objectives/evaluation of risk communication that is expected. Review studies, commentaries, conference proceedings, and books were excluded. Since abstracts were reviewed during the first round of eligibility assessment, materials that did not provide an abstract were excluded. Articles that discuss the procedures of future risk communication activities—meaning that the activities had not been implemented at the time of publication—were also excluded.

Search strategy

Potential materials for this study were identified on April 18, 2018, through relevant search engines, namely, PubMed, ScienceDirect, and PsycINFO. These sources are accepted as the world’s leading scholarly search engines since they provide access to peer-reviewed literature in a wide range of academic disciplines. The only search term used was “risk communication.” The Journal of Risk Research was also included as a source for the material collection because this journal contains many study articles on the topic of risk communication that are not covered by these search engines. Initially, the material search was not limited by the year of publication to learn how the number of publications related to risk communication activities has shifted over time. After eliminating duplicates and articles that did not provide an abstract, and obtaining a solid idea about the number of possibly eligible studies, the authors used this information to determine the span of publication years from which studies would be included. This time span includes approximately half of the relevant materials published in recent years.

Eligibility assessment

This study conducted two rounds of eligibility assessment. As it is briefly stated above, the first round was a review of only titles and abstracts of articles that were identified through the search engines and the Journal of Risk Research. This initial assessment was to pre-select materials from which to derive the eligibility criteria for this study and to obtain a broad picture of recent risk communication activities in order to finalize a plan for subsequent analysis. The second round involved a review of full texts of pre-selected articles to confirm their eligibility.

For the first round of eligibility assessment, the team established groups consisting of two researchers. Each researcher independently assessed assigned articles and determined whether the study (1) evaluated risk communication activities quantitatively, (2) assessed the objectives and/or effects of risk communication activities qualitatively, and (3) discussed the objectives and/or effects of risk communication activities based on prior experiences and/or existing scientific knowledge. For the second round, researchers were re-grouped, and a pair of researchers independently read assigned articles that they had not checked during the first round to confirm the eligibility of the articles and finalize the material selection.

The principal investigator of the research project (MM) coordinated this evaluation and selection process. MM checked all articles and developed a basic protocol for the eligibility assessment. In general, if both reviewers who checked a particular article agreed in their evaluation of its eligibility, the decision was accepted. When there was a disagreement, MM facilitated discussions between the researchers to achieve consensus. When necessary, MM reflected the points of agreement in the protocols to ensure consistency in evaluation.

Coding process

After completion of the first round of eligibility assessment, the research team discussed what data should be extracted and how the information should be labeled and coded. The team made decisions based on characteristics of risk communication activities learned from the first round of eligibility assessment, the international and national organizations’ statements on risk communication (Table 1) and other relevant literature, as well as individual researchers’ experience and expertise.

Researchers remained in the same group formed for the second round of eligibility assessment, and separately extracted data from each assigned article and coded it as follows:

• Evaluation approach: (1) quantitative, (2) qualitative, and (3) based on prior experience and/or existing scientific knowledge (see the criteria in “Eligibility Assessment”).

• Study field: (1) medicine, such as health and pharmaceutical realms, (2) food safety, (3) chemical substances (other than food safety matters), (4) nuclear and radiological disasters/emergencies, (5) other disasters/emergencies, (6) climate change, and (7) other.

• Timing when a risk communication intervention was implemented in line with the phases in the disaster management cycle: (1) non-crisis or pre-crisis, including non-specified, (2) crisis, and (3) post-crisis, including recovery phase.

• Target audience: (1) citizens (e.g., individual citizens, residents, unspecified persons, and citizen groups) or Non-Profit Organizations (NPOs), and (2) other (e.g., government, professionals, and companies).

• Communication type: (1) individual/small group communication (e.g., doctor–patient–family communication and family communication), and (2) large-group/mass communication.

• Objective/indicator: (1) knowledge increase, (2) communication satisfaction, (3) change in risk perception and concern alleviation, (4) reduction in psychological distress, (5) trust building, (6) decision making and behavior change (e.g., risk acceptance, risk avoidance, and risk management, such as avoidance of unhealthy foods, seeking healthcare, disaster mitigation and preparedness, and community partnerships; attitude toward behavior and behavioral intention were also included in this category), (7) self-efficacy improvement, and (8) other.

With regard to “intervention timing”, this study employed the three-stage approach proposed by Coombs (2012). The term “crisis” in this study refers to the definition proposed by the same scholar (Coombs, 2007) as, “a significant threat to operations that can have negative consequences if not handled properly.” The pre-crisis period involves the detection of warning signs relating to such crisis and prevention and/or preparedness. The crisis period concentrates on identifying the onset of a crisis, controlling the situation, and minimizing negative impacts. The post-crisis period concerns rehabilitation and full recovery from the crisis, evaluation of crisis management, and better preparation for future crisis (Coombs, 2012). With regard to “objective/evaluation indicator”, the researchers jointly determined how indicators should be classified by referring to the definitions and purposes of risk communication stated by the selected international and national organizations (Table 1).

Where applicable, multiple response categories were selected. If both reviewers who checked a particular article classified it the same way, the decision was accepted. When the two researchers differed, discrepancies were evaluated by a third researcher. When needed, the issues were discussed with MM until all concerned researchers reached an agreement on the article’s classification.

Examples of evaluation indicators were drawn for this paper. Specifically, one example for each indicator was taken from the field of medicine, and another was from other fields due to the generally large number of relevant medicine-related articles. Examples were chosen based on the frequency of citation assessed on May 10, 2019 through Google Scholar and the clarity of applied methods. Even if some frequently cited studies targeted multiple indicators, they were referred to for only one indicator among all the applicable indicators.

Statistical analysis

Data was entered into a Microsoft Office Excel spreadsheet. Excel was used to compute descriptive information on the collected data. Additionally, sets of Pearson’s chi-squared test with Yates’s continuity correction and Fisher’s exact test were conducted to examine the associations by study field and by objective/evaluation indicator. The statistical analyses were performed with studies that belonged to a single category of all the variables except for the variable of “objective/evaluation indicator”. “Nuclear and radiological disasters/emergencies,” “climate change,” and “other” from the study field variable were excluded because of their small size. For the same reason, the “crisis” group and “post-crisis” group were combined in the analysis on the associations by study field, and the “crisis” group was excluded in the analysis on the associations by objectives/evaluation indicators. For analyses involving study field, post hoc test (Aoki, 2010) was conducted to determine where differences occurred if an initial analysis identified a significant difference between study field and other variables. R (R Development Core Team, 2020) was used for the statistical analyses. Test results were considered significant at P < 0.05. P-value adjustment by Holm’s method was applied for multiple comparisons.

Search results

Figure 1 summarizes the flow of the material search and selection for this study. The database search found 3,710 articles, mostly from PubMed (57%). Of those, 1,433 articles went through the first round, and 412 moved on to the second round of eligibility assessment. In the end, 292 articles published between 2011 and 2017 remained for review.

Characteristics of risk communication studies

The data generated for this study is provided in Table S1. The table contains basic information from all 292 articles. Figure 2 shows a descriptive summary of the data.

More than 80% of the studies quantitatively evaluated own risk communication practices. Over 60% were related to medicine. Studies classified as “other” included those addressing human–wildlife conflicts (Lu et al., 2018) and traffic safety (Feenstra, Ruiter & Kok, 2014; Wu & Weseley, 2013), as well as studies that used a risk scenario or involved multiple risk domains to investigate effective means or to assess intrapersonal and other factors of risk communication (Dawson, Johnson & Luke, 2017; Poortvliet & Lokhorst, 2016). Five studies (2%) fell in multiple study fields. The vast majority of activities were implemented during a non-/pre-crisis phase (93%), and one study involved multiple phases. Over 90% of the studies targeted citizens/NPOs. Of those, 15 studies or 5% also approached other target groups, such as medical professionals and farmers. As for communication type, more than 60% were communications to a large group audience or the public. Of those, seven studies or 2% were also conducted in the form of individual/small group communication. Frequently-used objectives/evaluation indicators were “decision making and behavior change,” “change in risk perception and concern alleviation,” and “knowledge increase” (61%, 44%, and 39%, respectively). Examples of objectives/evaluation indicators are shown in Table 2.

Comparison between risk communication definitions and purposes and main objectives/evaluation indicators

The authors of this study identified “knowledge increase,” “change in risk perception and concern alleviation,” and “decision making and behavior change” as areas of focus in all fields; these objectives are also discussed in the definitions and purposes of most organizations and agencies (Fig. 2, Table 1). Here, “knowledge” is about the risks of concern and related risk management policies and actions. “Change in risk perception” primarily focused on guiding individuals’ subjective judgment of risk to align with available scientific evidence. Table 1 also illustrates that “reduction in psychological distress” does not generally appear in the selected organizations’ definitions and purposes of risk communication, which is consistent with this study; this topic was rarely addressed in the studies reviewed in this research (2%).

In contrast, “trust building” is frequently included among the proposed purposes of risk communication, whereas it was rarely addressed in the studies assessed in this research (8%). Further, while the organizations extended the target of risk communication in their statements to non-citizen parties, such as industries and media, those groups were rarely targeted in risk communication activities in the studies assessed in this research. Consequently, relevant indicators were not discussed in the study articles.

Differences by study field

Table 3 shows the frequency data, which is cross-tabulated with study field and other variables. In most fields of study, risk communication activities were conducted in a non-/pre-crisis phase, whereas nearly half of the risk communications in the field of nuclear and radiological disasters/emergencies were conducted in a non-/pre-crisis phase, and the remaining were in a post-crisis phase. The table shows that the majority of risk communication targets citizens/NPOs. At the same time, 29% in the field of chemical substances targeted other groups. In the medical field, half of the risk communications were conducted at an individual level or in a small group, whereas risk communication in other fields was conducted mainly in a larger group or to an entire population of interest.

Table S2 shows the results of the analyses between study field and other variables. The series of analyses found significant associations between study field and communication type (P < 0.01). Multiple comparisons suggested a significant difference pertaining to communication type between risk communications in the field of medicine from those in food safety (P < 0.05).

Differences by objective/evaluation indicator

Table 4 highlights the diversity in objectives and indicators of risk communication activities. For instance, the fields of food safety and other (i.e., non-nuclear/radiological) disasters and emergencies had a higher percentage in terms of risk communications aiming or addressing “trust building” (22% and 19%, respectively) than other fields, especially compared to the field of medicine (5%). The chemical substance field had a higher percentage (76%), and the field of nuclear and radiological disasters/emergencies had a lower percentage (33%) compared with other fields with regard to risk communications focusing on “decision making and behavior change.”

Table 4 also suggests some percentage differences in intervention timing, target audience, and communication type by objective/evaluation indicator. The main objectives and indicators of risk communications conducted in a non-/pre-crisis period were “knowledge increase,” “change in risk perception and concern alleviation,” and “decision making and behavior change” (39%, 45%, and 62%, respectively). “Decision making and behavior change” was a main indicator for risk communications conducted in a crisis period (57%). “Knowledge increase,” and “change in risk perception and concern alleviation” were the main indicators for risk communications conducted in a post-crisis period (83%, 50%, respectively). There was over 15% difference in “change in risk perception and concern alleviation” between risk communications targeting citizens/NPOs compared to risk communications targeting other groups (45% vs. 25%). The same was observed for “change in risk perception and concern alleviation” and “decision making and behavior change” between risk communications conducted at an individual/small group level and risk communications conducted in larger groups (32% vs. 49% and 71% vs. 56%, respectively).

The analyses (summarized in Table S3) revealed a significant association between study field and “trust building” (P < 0.05), between intervention timing and “change in risk perception and concern alleviation” (P < 0.05), between communication type and “change in risk perception and concern alleviation” (P < 0.05), and between communication type and “decision making and behavior change” (P < 0.05). Multiple comparisons did not find any significant difference between study fields in terms of “trust building.”