Analysis of factors influencing hookwire dislodgement in CT-guided hookwire localization: a retrospective study using variable importance analysis with a random forest model

Study design and patients

We reviewed medical records from January 2012 to May 2024 at one tertiary institution to investigate cases of CT-guided hookwire localization and subsequent VATS resection for lung nodules. We checked where the surgical records documented the success or failure of localization, and cases without documented localization outcomes were excluded from the study. Cases in which localization was performed on two nodules during a single surgery, followed by VATS resection for both nodules, were also included. In such instances, each nodule was considered as a separate case for the purposes of this study. For example, if a single patient underwent localization for two nodules, it was classified and analyzed as two separate cases (Fig. 1).

CT-guided localization technique

CT-guided localization is a technique performed in a CT room, where imaging is used to identify the position of the target nodule, and a hookwire is inserted adjacent to the nodule under CT guidance. During the localization procedure, a limited scan range of the chest was covered to visualize the target nodule. All localization was conducted with 64- or 128- slice multidetector CT scanners (SOMATOM definition Edge and SOMATOM sensation 64; Siemens Medical Solutions, Forchheim, Germany). The scan parameters were as follows: 100 mA, 100 kVp and pitch of 1–1.2. The scan data were reformatted with a 3.0 mm section thickness for the transverse images. A 7.5 cm or 10 cm length with 20 gauge size localization hookwire (Argon Localization Needle, Argon Medical, Plano, TX, USA) was used for nodule localization. The CT images with limited scan coverage are reviewed to determine the needle puncture site. Subsequently, local anesthesia is administered to the skin around the selected puncture site. The cannula needle housing the hookwire is then gradually inserted through the chest wall and into the lung parenchyma under sequential CT guidance. Once the desired position is reached, the outer cannula needle is withdrawn, releasing the hookwire’s horn, which anchors the wire securely in place. Finally, a CT image is obtained with the hookwire secured in its final position, allowing the thoracic surgeon to understand the relationship between the insertion site and the nodule before surgery, ensuring optimal surgical planning and execution.

Potential factor selection

Dislodgement occurring during the procedure was defined as the outcome. Potential predictor variables were selected considering procedure-related characteristics, radiologic features of patient findings, general characteristics, and in-hospital management factors. Procedure-related characteristics included distance from the nodule to the pleura (DNP), distance from the wire tip and the pleura (DWP), total depth, muscle depth, chest wall depth, multiple localization, presence of post-procedure pulmonary hemorrhage, presence of post-procedure pneumothorax, nodule penetration and procedure time. For an in-hospital management characteristic, PS interval was selected. General patient characteristics included age and sex, while radiologic features related to the patient’s lesion and lungs comprised nodule size, nodule subtype, and emphysema.

Procedure-related characteristics

CT images obtained during CT guided localization were reviewed by a thoracic radiologist (J.H.H.) who was blinded whether the case was of dislodgement or not. The maximum diameter of the nodule from skin (total depth), the thickness of the chest wall and muscle layer (chest wall depth and muscle depth) traversed by the localization hookwire, DWP, DNP were measured using electronic calipers (millimeters) (Fig. 2). Additionally, we checked whether the localization hookwire penetrated the nodule (nodule penetration). Post procedural pneumothorax and pulmonary hemorrhage were recorded. Pulmonary hemorrhage was defined as new consolidation or GGNs on post-procedural images and was recorded as presence of hemorrhage, including needle tract hemorrhage less than two cm in width (Tai et al., 2016). We also assessed cases performing multiple localizations during a single localization procedure (multiple localization). The initiation and completion time of the CT-guided localization procedure was reviewed to evaluate procedure time.

Patient-related characteristics

Sex and age were examined my medical record. Preoperative CT scans were reviewed by a thoracic radiologist (J.Y.K). The radiologist evaluated the nodule size and subtypes and location of the nodule. The nodule subtype was classified into three categories: pure GGNs, part-solid nodules (PSNs), and solid nodules (Sun et al., 2024). The degrees of emphysema (0, none; 1, trace or mild; 2, moderate; 3, confluent; 4, advanced destructive) was defined based on the CT-based Visual Classification of Emphysema, which has been used in various studies (Lynch et al., 2018).

In-hospital management related characteristics

The initiation of the CT-guided localization procedure, and the initiation times of the surgery were reviewed to evaluate PS interval.

Outcome

The presence of hookwire dislodgement in CT-guided hookwire localization was determined by reviewing operation notes (surgical record). The surgeon documented whether the wire localization was well positioned in the operation field. We retrospectively reviewed the operation notes, and cases where the documentation did not clearly state were excluded from the analysis (n = 24).

Statistical methods

The groups were classified based on the presence or absence of dislodgement. The characteristics of each group were then summarized by presenting the mean and standard deviation. We conducted Mann–Whitney test for comparing continuous variables among both groups, due to the small number of dislodgement cases (n = 15). Chi-squared test with Yates’s correction was performed for comparing categorical variables. PS interval (minutes), DNP (mm), DWP (mm), total depth (mm), age (years), procedure time (minutes), nodule size (mm), muscle depth (mm), and chest wall depth (mm) were analyzed as continuous variables. Nodule subtype (pure GGNs, PSNs, and solid nodules), sex (male/female), pulmonary hemorrhage (present/absent), emphysema (grade 0/1/2/3/4), multiple localization (yes/no), nodule penetration (yes/no) and pneumothorax (present/absent) were analyzed as categorical variables.

We aimed to analyze relative variable importance and rank the potential variables. Given the prevalence of categorical variables in the dataset, a random forest model was employed as an analytical method to assess the importance of each variable’s influence on the outcome variable. Random forest is an ensemble learning technique that enhances predictive performance by aggregating multiple decision trees. Each tree is trained on different samples, which increases the model’s robustness. The model is trained by repeatedly sampling a subset of the data with replacement (bootstrapping) to create multiple decision trees. Each tree in the forest is built from a different bootstrap sample, which typically includes about two-thirds of the original data. The remaining one-third of the data, known as out-of-bag (OOB) samples, which are not used in the training of each tree, allow for an assessment of the model’s accuracy and stability without the need for separate validation data (Breiman, 2001). The number of trees was set to 500, and the number of variables tried at each split was set to 4, which is the square root of the total number of predictors. The minimum size of terminal nodes was set to 1. The splitting criterion used for node division was the Gini Index. The Gini Index, which measures node impurity, was calculated as $Gini=1-\sum p_i^2$, where p_i is the proportion of samples in class i. For each node, the model selects the variable and splitting point that minimize the weighted Gini Index of the child nodes. Tree depth, representing the number of nodes from the root to the deepest leaf node, was not explicitly restricted during model construction, allowing the trees to grow until terminal nodes contained the minimum number of samples (one sample per node). Tree depth was presented to understand the complexity of the individual trees, and the average, maximum, and minimum depths were presented.

Due to the significant imbalance in the data distribution, with group sizes of 15 and 108, it was deemed inappropriate to classify the data using a random forest model directly. Thus, the Synthetic Minority Over-sampling Technique (SMOTE) method was applied to address this issue. SMOTE addresses data imbalance by synthesizing new samples for the minority class and adjusting the majority class through oversampling and undersampling (or random sampling), respectively. In SMOTE, oversampling of the minority class (the group with dislodgement) was performed using the K-nearest neighbors (KNN) algorithm. Specifically, KNN was employed to identify the K-nearest neighbors for each minority class data point, and new synthetic data points were generated by creating linear combinations of these neighbors and the existing data points (RColorBrewer & Liaw, 2018). In our implementation, the number of nearest neighbors was set to 5. For each minority class instance, synthetic samples were created by interpolating between the instance and five of its nearest neighbors. The minority class data was increased by 600%, and for the majority class (the group without dislodgement), random sampling with replacement was conducted to match 150% of the newly oversampled minority class.

The results of the random forest model were described using OOB (out-of-bag) data, focusing on the sensitivity, specificity, accuracy of the predicted values compared to the actual values, and the P-value (for accuracy > no information rate). The validation of the random forest model, trained on the SMOTE-oversampled dataset, was performed using 10-fold cross-validation, and the results were included in the analysis. Additionally, the model was applied to the original dataset without SMOTE, and the sensitivity, specificity, accuracy, and P-value were also described. The importance of variables, as results of random forest model, were presented using mean decrease accuracy (MDA) and mean decrease Gini (MDG). These metrics provide insights into the relative importance of each predictor in the model. MDA evaluates variable importance in random forest by measuring the drop in model accuracy when a variable’s values are randomly permuted. Larger decreases indicate higher importance, and MDA is calculated using OOB data. MDG measures the importance of each variable by calculating the decrease in the Gini impurity when the variable is used in the tree splits (Han, Guo & Yu, 2016).

As part of the sensitivity analysis, the variable importance ranking was re-evaluated using the eXtreme gradient boosting (XGBoost) method. The minimum sum of instance weights required in a child was set to 1. The maximum depth of the decision trees set to 6. The learning rate was set to 0.01. The number of boosting iterations was set to maximum 1,000, with early stopping enabled. The subsample rate was set to 0.8. Feature importance was evaluated using the Gain metric, which was calculated based on the XGBoost model. The importance of each variable was determined by assessing its contribution to the reduction in the model’s loss function during the tree-building process. The Gain represents the average improvement in model performance, where a higher Gain value indicates that the feature plays a more significant role in reducing prediction errors. Due to the nature of XGBoost, which accepts input values in the form of a numeric matrix, variables with three or fewer categories were converted into dummies. For the three-category variable, nodule subtype, the reference variable (“00” coding) was set as pure GGN. Variables with more than three categories, such as emphysema grade, were encoded using one-hot encoding.

R project 4.4.0 (https://r-project.org) was used for statistical analysis. Package “randomForest” was used for random forest modelling, “caret” for k-fold validation of random forest, “DMwR” for SMOTE, “xgboost” for XGBoost modeling.

Study participants

A total of 141 patients were identified, of which 24 were excluded due to unclear documentation of localization dislodgement in the surgical records. Six patients underwent two localization procedures simultaneously. A total of 123 procedures were performed on 117 patients.

General characteristics

The general characteristics of the participants are presented in Table 1. Among 123 included cases, 15 cases (12.2%; M:F, 7:8) showed wire dislodgement. The mean age was 67.2 ± 14.0 years in the dislodgement group and 66.0 ± 10.7 years in the non-dislodgement group. Of these dislodged 15 cases, eight nodules were located in the lower lobes (right lower lobe, 5; left lower lobe, 3), and seven were in the upper or middle lobes (right upper lobe, 5; right middle lobe, 1; left upper lobe, 1). The mean nodule size was 11.7 mm (range, five mm to 28 mm). Seven cases were solid nodules, seven were PSNs and one case was GGN. Among these, seven nodules were penetrated with the hookwire, while eight were not. Age, sex, emphysema degree, nodule subtypes (solid, GGNs or PSNs) and nodule size were not statistically significant differences between the dislodgement and non-dislodgement groups (Table 1). Despite the dislodgement, resections of all 15 cases were successfully performed, although two cases were converted to lobectomy instead of wedge resection. Both cases were confirmed as lung adenocarcinoma, and lobectomy was the appropriate treatment for them.

During the localization procedure, parenchymal hemorrhage was observed in 53 cases (43.0%). Pneumothorax occurred in 61 cases (49.0%). In the dislodgement group, five cases (33.3%) of pneumothorax and nine cases (60.0%) of parenchymal hemorrhage occurred. In the non-dislodgement group, 56 cases (51.9%) of pneumothorax and 44 cases (40.7%) of parenchymal hemorrhage occurred. No statistical significant difference were observed in both groups according to post procedural complications. None of the patients required specific intervention associated for post-procedural complications.

The PS interval was significantly different between dislodgement and non-dislodgement group, with 134.1 ± 73.1 min in the dislodgement group and 104.1 ± 46.1 min in the non-dislodgement group (p = 0.031). However, total depth (58.4 ± 13.6 mm and 60.2 ± 19.2 mm, respectively), DNP (16.3 ± 11.2 mm and 16.1 ± 14.3 mm, respectively), DWP (24.6 ± 15.5 mm and 31.4 ± 14.0 mm, respectively), procedure time (19.3 ± 17.5 min and 15.3 ± 7.1 min, respectively), chest wall depth (41.7 ± 12.0 mm and 41.1 ±  11.9 mm, respectively), muscle depth (18.9 ± 11.2 mm and 19.0 ± 11.0 mm, respectively) were not significantly different between the two groups (Table 1). Presence of nodule penetration and multiple localization were also not statistically different (dislodgement versus non-dislodgement, penetrated, seven cases, 46.7% versus 55 cases, 50.9%; presence of multiple localization, four cases, 26.7% versus eight cases, 7.4%).

The performance of the random forest model

Using the SMOTE for oversampling, the dataset was constructed with 105 cases in the dislodgement group (15 original cases and 90 synthetic cases) and 135 cases in the non-dislodgement group (random sampling with replacement from 108 original cases). The characteristics of the dataset after performing SMOTE are provided in (Table S1).

The tree depths ranged from a minimum of seven to a maximum of 16, with an average depth of 10.1. The error rate performed on OOB data was 5.8%. Among the 240 total cases including synthetic data, the model correctly predicted dislodgement occurrence in 96 cases (true positives) and correctly identified non-dislodgement in 130 cases (true negatives). It failed to predict dislodgement in nine cases (false negatives) and incorrectly predicted dislodgement in five cases (false positives), resulting in a sensitivity of 0.91 (95% CI [0.92–0.99]) and a specificity of 0.96 (95% CI [0.84–0.96]). The classification accuracy was 0.94 (95% CI [0.90–0.98]), and the P-value (for accuracy > no information rate) was <0.0001, indicating significance. The negative predictive value was 0.93 (95% CI [0.88–0.97]), and the positive predictive value was 0.95 (95% CI [0.89–0.98]). The F1 score was 0.95, and the Kappa coefficient was 0.88. The results of the 10-fold cross-validation performed on the oversampled dataset showed an accuracy of 0.95 (95% CI [0.92–0.98]) and a Kappa value of 0.90 (95% CI [0.84–0.96]). Considering the model was built with synthetic data, another validation was performed on the original dataset (15 dislodgement cases and 108 non-dislodgement cases). Of the cases predicted to be positive (group where dislodgement occurred), 22 cases were identified, with 15 being actual cases of dislodgement and seven being false positives. In the group predicted to be negative (group where dislodgement did not occur), 101 cases were identified, with no false negatives. The classification accuracy was 0.94 (95% CI [0.89–0.98]), and the P-value (for accuracy > no information rate) was 0.01, indicating significance. Assuming dislodgement cases as positive, the model demonstrated a specificity of 0.93 (95% CI [0.87–0.97]) and a sensitivity of 1.00 (95% CI [0.78–1.00]). The negative predictive value was 1.00 (95% CI [0.96–1.00]), while the positive predictive value was 0.68 (95% CI [0.45–0.86]). The Kappa coefficient was 0.78.

Analysis of variable importance order using a random forest model

In the random forest model, variable importance was assessed using mean decrease accuracy and mean decrease Gini criteria. The most important variable was PS interval, followed by DWP, DNP, total depth, and age. A sharp decline in importance was observed between the top five variables and the sixth variable. Variables such as chest wall depth, procedure time, muscle depth, nodule size, and nodule subtype formed a subsequent group, with another decline in importance observed at the 11th variable. The bottom five variables for both criteria were multiple localization, sex, hemorrhage, emphysema, pneumothorax, and presence of nodule penetration (Fig. 3).

Analysis of variable importance order using a XGBoost model

The sensitivity analysis using XGBoost for variable importance ranking also identified PS interval as the most significant variable explaining dislodgement. Following this, age, DWP, total depth, and DNP were ranked in descending order of importance. While there were differences in the specific order, the top five variables identified were consistent with those from the random forest model. Variables ranked after the top five included nodule subtype (solid nodule), sex, and wall depth in descending order. The graph illustrating variable importance derived from the XGBoost analysis is provided as a Fig. 4.