Prediction of falls using a risk assessment tool in the acute care setting
Department of Medicine, McMaster University, Hamilton, Ontario, Canada
School of Rehabilitation Sciences, McMaster University, Hamilton, Ontario, Canada
Department of Statistics & Actuarial Sciences, University of Waterloo, Waterloo, Ontario, Canada
Department of Clinical Health Sciences, McMaster University, Hamilton, Ontario, Canada
Innovus Research Inc., Burlington, Ontario, Canada
Hamilton Health Sciences, Hamilton, Ontario, Canada
Abstract
Background
The British STRATIFY tool was previously developed to predict falls in hospital. Although the tool has several strengths, certain limitations exist which may not allow generalizability to a Canadian setting. Thus, we tested the STRATIFY tool with some modification and re-weighting of items in Canadian hospitals.
Methods
This was a prospective validation cohort study in four acute care medical units of two teaching hospitals in Hamilton, Ontario. In total, 620 patients over the age of 65 years admitted during a 6-month period. Five patient characteristics found to be risk factors for falls in the British STRATIFY study were tested for predictive validity. The characteristics included history of falls, mental impairment, visual impairment, toileting, and dependency in transfers and mobility. Multivariate logistic regression was used to obtain optimal weights for the construction of a risk score. A receiver-operating characteristic curve was generated to show sensitivities and specificities for predicting falls based on different threshold scores for considering patients at high risk.
Results
Inter-rater reliability for the weighted risk score indicated very good agreement (inter-class correlation coefficient = 0.78). History of falls, mental impairment, toileting difficulties, and dependency in transfer / mobility significantly predicted fallers. In the multivariate model, mental status was a significant predictor (P < 0.001) while history of falls and transfer / mobility difficulties approached significance (P = 0.089 and P = 0.077 respectively). The logistic regression model led to weights for a risk score on a 30-point scale. A risk score of 9 or more gave a sensitivity of 91% and specificity of 60% for predicting who would fall.
Conclusion
Good predictive validity for identifying fallers was achieved in a Canadian setting using a simple-to-obtain risk score that can easily be incorporated into practice.
Background
Falls account for at least 40% of all accidents in hospital
Clinical prediction rules are tools designed to predict health outcomes and assist health care professionals plan patient care. These tools typically include three or more risk factors from a patient's history or physical exam that predict an outcome such as falls
However the STRATIFY tool has limitations. Altman
We tested the predictive validity of the STRATIFY variables in a Canadian setting, with the objective of determining if it predicts fallers rather than patient falls. We constructed a weighted risk score based on the components of the STRATIFY tool and examined its predictive validity.
Methods
Setting and participants
Data were collected over a 6-month period in 2000 at Hamilton Health Sciences, Hamilton, Ontario, Canada, a multi-site teaching hospital. Patients over 65 years of age admitted consecutively to four general medical units having a total of 114 beds were assessed. Palliative and critical care patients were excluded.
Definition of clinical outcome
The clinical outcome we studied was "fallers" versus "non-fallers". "Fallers" were those having one or more falls during hospitalization. A fall was defined as an individual involuntarily coming to rest on the ground or surface lower than their original station
Predictive variables: identification and definition
Five patient characteristics found to be risk factors for falls in the British STRATIFY study
Table 1
Odds ratio (OR) and 95% confidence intervals (CI) from the univariate and multivariate logistic regression models.
Item
Univariate Analysis
Multivariate Analysis
OR
95% CI
P-value
OR
95% CI
P-value
History of falls
2.47
(1.23,4.96)
0.011
1.88
(0.91,3.89)
0.089
Mental Status
6.07
(2.86,12.86)
<0.001
4.06
(1.81,9.16)
<0.001
Vision Impaired
1.02
(0.51, 2.05)
0.946
1.09
(0.53,2.24)
0.824
Toilet Difficulty
2.79
(1.37,5.67)
0.005
1.20
(0.52,2.75)
0.669
Transfer / Mobility
3.71
(1.83,7.50)
<0.001
2.10
(0.92,4.78)
0.077
Additional File 1
Appendix I: Falls Risk Screening Tool and Sample Calculations. Appendix I: Falls Risk Screening Tool and Sample Calculations. A description of how the weighted risk score formula was derived is described in this file. The falls risk screening tool is provided, with a patient example demonstrating how to calculate the overall risk score.
1741-7015-2-1-S1.doc
Screening assessment scoring
The nurses completing the screening tool had a 10-min orientation session run by the clinical nurse specialists who were part of the investigative team. The assigned nurse collected screening variables 24 to 48 hours after the patient was admitted to hospital in a 5-min bedside session.
To calculate predictive validity, each of the five items were scored dichotomously as 1 = present, and 0 = absent for items 1 to 4. The risk factors were deemed to be "present" if one or more of the statements were considered within each domain (Table
Statistical analyses
Each of the five items was assessed individually and collectively for their predictive power based on logistic regression models. The variables were also used to produce an overall risk score using two approaches. First, an unweighted risk score (r) was computed by simply counting the number of risk factors present. This gave a risk score ranging from 0 to 5, as in the original STRATIFY study.
Second, a weighted risk score was obtained based on the regression coefficients from the multivariate logistic regression model in which the outcome was the fall status of the patient. Specifically, the relative magnitude of the beta coefficients from the multivariate logistic model reflects the relative prognostic strength of the risk factors when they are jointly considered. Therefore the relevant information for the construction of the weights is the relative size of the beta coefficients and any weights, which preserve these ratios, will preserve good predictive validity (see Appendix [
To consider predictive validity in terms of sensitivity and specificity one must consider two populations, "fallers" and "non-fallers". Two receiver operating characteristic curves (ROC) were constructed to display the varying specificity and sensitivity values applicable to the range of possible thresholds of unweighted (0 – 5) and weighted risk (0 – 30) scores that could be used to classify patients as high risk for falling or not. All analyses were performed using the Statistical Analysis System (SAS release 8.1).
Inter-rater reliability
Two nurses independently assessed a sub-sample of 35 patients. The order of assessments was random and assessors were blind to the findings of the other nurse. The required sample size was estimated at 33 patients, with power = 0.80 and alpha = 0.05 using published power tables
Results
Over 6 months, 620 patients were screened for falls. The mean age was 78 years (SD 7.7) and 338 (54.5%) patients were female. The diagnoses most responsible for hospitalization in the sample were circulatory disorders (45.2%), respiratory disorders (20.8%), digestive disorders (4.0%) and mental disorders (2.9%). Diagnosis was not predictive of falls. Thirty-four patients (5.5%) fell at least once during their hospitalization and there were a total of 77 falls. In total, 171 patients (27.6%) had a history of falls at admission.
Predictive variables
Based on univariate logistic regression (Table
When length of stay was added to the multivariate model, the odds ratios associated with each risk factor were slightly attenuated. However, the pattern of results remained the same and mental status was most predictive, followed by history of falls, and then transfer/mobility difficulties.
A weighted risk formula (Appendix I) was derived which shows the relative magnitude of the regression coefficients from the multivariate logistic regression model. This formula can easily be used to calculate the weighted risk score.
ROC curves
The ROC curves based on the unweighted and weighted risk scores are given in Figure
Figure 1
Receiver operating characteristic curve (ROC) for weighted (0 – 30) and unweighted (0 – 5) risk score
Receiver operating characteristic curve (ROC) for weighted (0 – 30) and unweighted (0 – 5) risk score
Table 2
Sensitivities and specificities for threshold of weighted risk score (0 – 30)
Risk Score
False Positive Rate
False Negative Rate
Sensitivity
Specificity
0
100.0
0.0
100.0 (100.0,100.0)
0.0 (0.0, 0.0)
1
81.7
2.9
97.1 (91.4,102.7)
18.3 (15.1,21.4)
2
61.3
5.9
94.1 (86.2,102.0)
38.7 (34.8,42.7)
3
59.0
5.9
94.1 (86.2,102.0)
41.0 (37.0,44.9)
6
52.9
8.8
91.2 (81.6,100.7)
47.1 (43.1,51.1)
7
50.5
8.8
91.2 (81.6,100.7)
49.5 (45.4,53.5)
8
43.0
8.8
91.2 (81.6,100.7)
57.0 (53.0,61.0)
9
39.8
8.8
91.2 (81.6,100.7)
60.2 (56.3,64.2)
10
34.1
14.7
85.3 (73.4, 97.2)
65.9 (62.0,69.7)
13
32.3
20.6
79.4 (65.8, 93.0)
67.7 (64.0,71.5)
14
31.4
23.5
76.5 (62.2, 90.7)
68.6 (64.8,72.4)
15
27.5
29.4
70.6 (55.3, 85.9)
72.5 (68.9,76.1)
16
24.7
32.4
67.6 (51.9, 83.4)
75.3 (71.8,78.7)
17
22.7
38.2
61.8 (45.4, 78.1)
77.3 (73.9,80.7)
20
21.2
44.1
55.9 (39.2, 72.6)
78.8 (75.5,82.1)
21
19.6
52.9
47.1 (30.3, 63.8)
80.4 (77.2,83.6)
22
17.1
61.8
38.2 (21.9, 54.6)
82.9 (79.9,86.0)
23
15.2
67.6
32.4 (16.6, 48.1)
84.8 (81.9,87.7)
24
8.2
73.5
26.5 (11.6, 41.3)
91.8 (89.6,94.0)
27
4.6
79.4
20.6 (7.0, 34.2)
95.4 (93.7,97.1)
28
4.1
79.4
20.6 (7.0, 34.2)
95.9 (94.3,97.5)
29
3.8
79.4
20.6 (7.0, 34.2)
96.2 (94.7,97.8)
30
2.2
85.3
14.7 (2.8, 26.6)
97.8 (96.6,99.0)
The ROC curve for the un-weighted risk score (r) (Figure
Inter-rater reliability
The inter-rater reliability between the two nurses using the weighted risk score gave an ICC of 0.78 (95% CI: 0.60, 0.88). The reliability of the binary classification of high / normal risk gave kappa = 0.63 (95% CI: 0.36, 0.90). The unweighted risk score gave a similar reliability (ICC = 0.68; 95% CI: 0.46, 0.83).
Discussion
Falls in the elderly are often a symptom of acute medical problems in combination with underlying risks such as medications, postural hypotension and lower extremity weakness. Identifying those at risk allows targeted assessment and intervention such as a review of medications and environmental modifications
Risk factors included in screening for falls in hospitalized patients have largely been consistent across studies with varying methods. Findings have repeatedly emphasized falls history, mental impairment, toileting frequency, and general mobility as predictive variables for falls
Studies have also differed in the suggested ideal risk score cut-offs to consider patients in the "at risk" group. Whether our suggested cut-off of 9 is ideal for different hospital settings is not known. One approach is to use the cut-off that maximizes predictive power mathematically. Practitioners in different settings may adjust the trade-off between sensitivity and specificity, based on differing falls rates, values, laws, funding and other factors.
Our finding of poor predictive validity with the unweighted items does not clearly amount to poor generalization across settings because the items and protocol were changed. Studies involving tests of prediction tools in new locations have found results that are weaker than the original findings
A potential methodological limitation is uncertainty about patient incident reports, which may not capture all falls, however our documented rate of falls was similar to previous years in our setting. There is also the possibility that completing risk assessments influences how nurses respond to patients in terms of falls prevention strategies (i.e. Hawthorne effect). It is unknown if this factor affected the true rate of falls in our setting. However, it is predicted that this effect is likely minimal given that strong consistent falls prevention strategies were not in place at the time of the study. Another potential limitation is that despite some changes to scale items taken from STRATIFY to improve reproducibility; there is still room for error. For example, patient recall of falls may not be accurate and consistent. This may have accounted for an inter-rater reliability that was lower than ideal. The problem stresses a potential need to improve operational definitions of risk variables to ensure reproducibility in measuring items such as mental status. It may help to have consensus among investigators on key issues including: which variables to include, operational definitions of risk constructs, the duration over which risk is assessed (i.e. within 24 or 48 hours), the way in which users should be trained, and what the appropriate outcomes are (e.g. falls versus fallers). Further replication of our study in other settings will help to correct upon these limitations and improve generalizability.
Conclusions
We found good predictive validity for identifying fallers in hospital in a Canadian setting after weighting risk factors. Assessment of fall risk is one of the essential quality indicators for care of the older adult.
Competing Interests
None declared.
Authors' contributions
AP, WP, JA and EC were involved with study design. AP, WP, EC and NF were involved with study development, co-ordination and data collection. AP, NF and RC were involved with data analysis. All authors were involved in preparing and writing the manuscript and approved the final version.
Acknowledgements
We wish to thank Tish Butson and Cathy Reiss from the Hamilton Health Sciences for their assistance in study nurse orientation and co-ordination of the study. This study was supported by a grant through the Hamilton Health Sciences Foundation. Dr. A Papaioannou is an Ontario Ministry of Health Career Scientist. Dr. RJ Cook is an investigator for the Canadian Institutes for Health Research.
Accidental injuries of hospitalized patients. A prospective cohort study
Hip fracture frequent in hospital
Hospital falls: Development of a predictive model for clinical practice
Falls risk factors in an acute-care setting: a retrospective study
Risk factors for falls in hospitalized older medical patients
Clinical prediction rules: a review and suggested modifications of methodological standards
A 6-point risk score predicted which elderly patients would fall in hospital
Study to predict which elderly patients will fall shows difficulties in deriving and validating a model
Statistics Notes: Units of analysis
Predicting ward falls. Validation of a quick nurse-led assessment of falls risk
Hip fracture frequent in hospital
Inpatient accidents in hospitals
The Barthel ADL index: a reliability study
Sample size requirements for reliability studies
Quality indicators for the management and prevention of falls and mobility problems in vulnerable elders
Fall risk assessment measures: an analytic review
Initiation and Evaluation of a Research-based fall prevention program
Improving safety for hospitalized elderly
Managing falls: Identifying population-specific risk factors and prevention strategies
Screening older adults at risk of falling with the Tinetti balance scale
Reducing patient falls: A research-based comprehensive fall prevention program
Evaluation and implementation of a research-based falls assessment innovation
A prospective study to identify the fall prone patient
Evaluation of the STRATIFY falls prediction tool on a geriatric unit