J Cancer 2019; 10(5):1263-1274. doi:10.7150/jca.32386 This issue Cite

Research Paper

Development and Validation of Nomograms Predictive of Axillary Nodal Status to Guide Surgical Decision-Making in Early-Stage Breast Cancer

Jiao Li1†, Weimei Ma1†, Xinhua Jiang1, Chunyan Cui1, Hongli Wang2, Jiewen Chen1, Runcong Nie1, Yaopan Wu1 Corresponding address, Li Li1 Corresponding address

1. Department of Medical Imaging, Sun Yat-sen University Cancer Center, State Key Laboratory of Oncology in South China, Guangzhou, People's Republic of China
2. Breast Tumor Center, Sun Yat-sen Memorial Hospital, Sun Yat-sen University, Guangzhou, Guangdong, People's Republic of China
† Equal contributors

Citation:
Li J, Ma W, Jiang X, Cui C, Wang H, Chen J, Nie R, Wu Y, Li L. Development and Validation of Nomograms Predictive of Axillary Nodal Status to Guide Surgical Decision-Making in Early-Stage Breast Cancer. J Cancer 2019; 10(5):1263-1274. doi:10.7150/jca.32386. https://www.jcancer.org/v10p1263.htm
Other styles

File import instruction

Abstract

Purpose: To develop and validate nomogram models using noninvasive imaging parameters with related clinical variables to predict the extent of axillary nodal involvement and stratify treatment options based on the essential cut-offs for axillary surgery according to the ACOSOG Z0011 criteria.

Materials and Methods: From May 2007 to December 2017, 1799 patients who underwent preoperative breast and axillary magnetic resonance imaging (MRI) were retrospectively studied. Patients with data on axillary ultrasonography (AUS) were enrolled. The MRI images were interpreted according to Breast Imaging Reporting and Data system (BI-RADS). Using logistic regression analyses, nomograms were developed to visualize the associations between the predictors and each lymph node (LN) status endpoint. Predictive performance was assessed based on the area under the receiver operating characteristic curve (AUC). Bootstrap resampling was performed for internal validation. Goodness-of-fit of the models was evaluated using the Hosmer-Lemeshow test.

Results: Of 397 early breast cancer patients, 200 (50.4%) had disease-free axilla, 119 (30.0%) had 1 or 2 positive LNs, and 78 (19.6%) had ≥3 positive LNs. Patient age, MRI features (mass margin, LN margin, presence/absence of LN hilum, and LN symmetry/asymmetry), and AUS descriptors (presence of cortical thickening or hilum) were identified as predictors of nodal disease. Nomograms with these predictors showed good calibration and discrimination; the AUC was 0.809 for negative axillary node (N0) vs. any LN metastasis, 0.749 for 1 or 2 involved nodes vs. N0, and 0.874 for ≥3 nodes vs. ≤2 metastatic nodes. The predictive ability of the 3 nomograms with additional pathological variables was significantly greater.

Conclusion: The nomograms could predict the extent of ALN metastasis and facilitate decision-making preoperatively.

Keywords: Breast cancer, Lymph node, Magnetic resonance imaging, Metastasis, Ultrasonography

Introduction

Sentinel lymph node biopsy (SLNB) has been replacing axillary lymph node dissection (ALND) as the method of choice among early breast cancer (EBC) patients with clinically negative axillary disease (1,2). The American College of Surgeons Oncology Group Z0011 trial showed that SLNB without ALND in patients with clinical T1/2 N0 cancers did not compromise survival in cases with ≤ 2 metastatic sentinel lymph nodes (SLN) (3,4). Although SLNB represents major progress in the field of surgery and can avoid unnecessary ALND, a discussion on the disadvantages associated with SLNB is important. Studies have reported that, approximately 60% of patients with positive SLNs have no residual disease after ALND (5). Furthermore, the assessment of the pathological state of SLNs during surgery is both time-consuming and expensive (6,7). SLNB is also associated with injuries such as upper limb edema, arm numbness (8), weakened shoulder, and reduced arm strength (9). In fact, most patients with early breast cancer have node-negative disease, and SLNB can be avoided if reliable preoperative diagnostic evaluation of the axilla is available (10). Accordingly, axillary ultrasound (AUS) has become a routine preoperative evaluation for predicting axillary nodal status in many institutions (11,12). However, the diagnosis of nodal metastasis via ultrasonography is operator-dependent, and not very sensitive in cases with minor axillary metastatic burden (11,13,14). Other imaging modalities, such as conventional magnetic resonance imaging (MRI), positron emission tomography, and computed tomography (CT), are believed to be supplementary to SLNB in terms of diagnostic performance (15-18). However, conventional CT and radionuclide imaging of the lymph node (LN) are preferred in patients with advanced disease. A potential advantage of MRI over ultrasound (US) is that it offers a more global view of the axilla (19). At present, dynamic contrast-enhanced MRI is a widely used modality for breast mass characterization, breast cancer staging and prognostic parameter correlation. Previous investigators have mainly evaluated axillary MRI, rather than considering the primary tumor imaging characteristics and biology of the primary breast tumor (20,21). Nevertheless, the ALN status reflects the timeline of tumor development, and hence, assessing tumor characteristics may be important to guide the extent of axillary surgery (10).

Considering the pooled impact of radiological features with clinicopathological data, the feasibility of identifying the outcome of ALN involvement in EBC patients remains unclear. Based on the results of the Z0011 trial and previous literature (10), the clinical study endpoints were divided into 3 parts: no axillary lymph node involvement (N0), 1 or 2 nodal metastases (N+(1-2)), and ≥3 metastatic lymph nodes (N+(≥3)). Here, we aimed to evaluate the axillary status in EBC patients with clinically negative axilla preoperatively to determine whether treatment should involve no axillary staging, SLNB or completion ALND.

Materials and methods

Patient enrollment

The institutional review board approved the retrospective study, and the need for informed consent was waived. Patients with primary breast tumors who underwent preoperative breast and axillary MR examination between May 2007 and December 2017 were initially considered eligible for this study (n = 1799), and their electronic medical records were reviewed. The exclusion criteria were as follows: benign lesions, lobular or ductal carcinoma in situ, and other malignant breast tumors such as lymphoma and sarcoma, clinical T3 or higher cancers or node-positive tumors on physical examination, bilateral or occult breast cancer, a history of prosthesis implantation, ipsilateral breast or axillary surgery for breast cancer, neoadjuvant chemotherapy, previous breast biopsy for breast lesions prior to MRI, and cytologically confirmed nodal metastasis. The remaining 397 patients who had clinical T1 or T2 invasive breast cancers with non-palpable ALNs were included in the study cohort. Among the 397 EBC patients who underwent preoperative breast and axillary MRI, 212 (53.4%) had positive LNs (133 with 1 or 2 positive ALNs and 79 with at least 3 positive ALNs) and 201 (50.6%) had negative ALNs (Fig. 1).

MRI examination protocol

Breast and axillary MR examinations were performed prior to ALN surgery with a 3.0 Tesla MRI system (DISCOVERY MR750 24LX, GE Medical Systems, Milwaukee, WI, USA) using an 8-channel vibrant breast coil, with the patient in the prone position and the ipsilateral arm elevated with close contact between the coil and axilla. The imaging protocol was as follows:

(a) Axial T1-weighted, fast-spin echo (FSE) sequence without fat suppression (repetition time, 802 ms; echo time, 6.96 ms; flip angle, 111°; slice thickness, 5 mm; matrix, 384 × 288; number of averages, 0.5; field of view, 340 × 340 mm2; NEX, 1.0; in-plane resolution, 0.88 × 1.18 mm2; and total imaging time, 2 min and 47 s). (b) T2-weighted ideal sequence with fat suppression (repetition time, 3150 ms; echo time, 49 ms; flip angle, 111°; number of signals acquired, 1; image coverage, 210 mm; acquisition section thickness, 5.0 mm; NEX, 3.0; field of view, 340 × 340 mm2; matrix, 384 × 256; in-plane resolution, 0.88 × 1.32 mm2; and total imaging time, 4 min and 26 s); and (c) dynamic contrast-enhanced vibrant sequence with coronal section reconstruction (thickness, 1.0 mm): repetition time, 3.9 ms; echo time, 1.7 ms; flip angle, 5°; field of view, 300-350 mm; acquisition matrix, 320 × 320; NEX, 0.71; and slice thickness, 1.4 mm with no inter-slice gap. An intravenous injection of Gd-DTPA (Magnevist; Schering, Berlin, Germany) was administered at a dose of 0.1 mmol/kg of body weight, at a rate of 3 mL/s with an MRI-compatible power injector (Mississippi XD2000, Ulrich, UIM, Germany). Contrast-enhanced image acquisition was initiated immediately after saline injection. The sequence was repeated 8 times without any time gaps. Each sequence lasted approximately 59s. A series of 9 sequences was used to determine the early-enhancement rate and contrast enhancement curve with a SUN ADW 4.3 workstation (General Electric, Milwaukee, WI, USA).

MRI analysis

All breast MRI images were reviewed by 2 breast radiologists (H.L.W. [reader 1] and X.H.J. [reader 2], with 5 and 10 years of experience, respectively) independently, based on the 2013 Breast Imaging Reporting and Data system (BI-RADS) lexicon; the readers were blinded to the histopathological findings. Any disagreement between the radiologists was resolved via consensus. The 2 radiologists determined the amount of fibroglandular tissue (FGT; almost entirely fat, scattered, heterogeneous, or extreme dense); background parenchymal enhancement (BPE; minimal, mild, moderate, or marked); the presence of unifocal, multifocal or multicentric disease; tumor size; mass/non-mass characteristics; tumor location; morphologic features; distribution; and internal enhancement patterns of each lesion (mass and non-mass enhancement). BPE was determined based on the extent of FGT enhancement in initial contrast-enhanced or subtraction images. In patients with multiple lesions, only the lesion with the largest diameter was evaluated. Multifocal or multicentric disease was considered in cases with suspicious findings for additional sites of malignancy within the same or different quadrant of the ipsilateral breast. Primary tumor size was measured along the longest diameter. Time-signal intensity curves of the primary tumors were determined using a SUN ADW 4.3 workstation. Quantitative assessment was performed using contrast T1-weighted MRI data by drawing a region of interest (ROI) with an electronic cursor around the inner margin of the primary tumor and creating time-intensity curves (TICs); the kinetic curve pattern was then categorized into 3 types (persistent, plateau, or washout) (Fig. 2A,B).

 Figure 1 

Flowchart of the study population. SLNB, sentinel lymph node biopsy; ALND, axillary lymph node dissection; N0, free-disease axilla; N+(1-2), lymph node metastasis involving 1 or 2 positive nodes; N+(≥3), lymph node metastasis involving at least 3 positive nodes.

J Cancer Image
 Figure 2 

Right axillary metastatic lymph node in a 27-year-old woman (T2, invasive ductal cancer, grade 2). (A) A high-signal tumor on the axial dynamic contrast-enhanced magnetic resonance imaging (MRI) map (green circle); (B) Type III time intensity curve (wash-out) of a right breast tumor; (C) Axial T1-weighted fast-spin echo sequence without fat suppression (black arrow) and (D) T2-weighted ideal sequence with fat suppression images show an enlarged lymph node (white arrow) with irregular margin and the absence of a hilum; (E) Coronal contrast-enhanced image clearly shows asymmetric lymph nodes (white arrow) at level 1 of the right axilla.

J Cancer Image

One radiologist (C.Y.C, with 10 years of experience in breast lymph node assessment [reader 3]) reviewed the T1-weighted without fat suppression sequence, T2-weighted fat-suppression sequence, and reconstructed coronal contrast-enhanced images independently (Fig. 2C,D,E). The radiologist was blinded to the results of surgical N-staging and assessed the ipsilateral axillary lymph nodes on MRI using predefined quantitative descriptors, including the long diameter (LD), short diameter (SD), LD/SD, LN margin, hilum, and symmetry; the targeted node was selected as it was the largest or most commonly suspected for metastasis in the ipsilateral axilla. Asymmetry refers to asymmetric lymph nodes in terms of number or size, in comparison with the contralateral side of the axilla in breast cancer, which was optimally evaluated using a coronal reconstructed sequence (17) (Fig. 2E).

US protocol and analyses

A total of 331 patients underwent preoperative axillary ultrasound to assess the morphological features of ipsilateral LN using the IU22 (PHILIPS, The Netherlands) and ACUSON S2000 (SIEMENS, Germany) systems, with a high-frequency transducer (12 to 15 MHz). US electronic reports were reviewed, and characteristics of ALNs on US, including ALN margin, cortical thickening, and absence/presence of the medulla and hilum were recorded.

Pathological evaluation

The clinical parameters and histopathologic information extracted from the electronic medical records included age, clinical T-staging, histological type, Nottingham histological grade, estrogen receptor (ER) and progesterone receptor (PR) status, human epidermal growth factor receptor 2 (HER2) status based on immunohistochemistry and in situ hybridization, Ki-67 status (threshold >14%), molecular subtype, and vascular invasion. ER and PR statuses were considered positive if the immunostaining was positive in >1% of tumor cells. HER2 positivity was defined as a score of 3+ on immunohistochemistry or amplification on fluorescence in situ hybridization (FISH) (22,23). Tumors with scores of 0 or 1+ were considered HER2-negative. Tumors that scored 2+ were evaluated further via FISH. If the ratio of the HER2 gene signal to chromosome 17 probe signal was more than 2.2, it was classified as HER2-positive (24). When HER2 with a score of 2+ lacked amplification on FISH, the molecular type was classified as uncertain.

Lymph nodes were surgically removed via SLNB or ALND. All nodes were examined postoperatively with serial section hematoxylin-eosin (H&E) staining. The presence of macrometastases (size of the largest metastases >2 mm cancer foci) or micrometastases (0.2-2 mm cancer foci) on SLNB was indicative of axillary node-positive status. Patients were considered free of ALN metastasis after negative findings on SLNB; however, if SLN metastasis was present, these patients routinely underwent completion ALND. Isolated tumor cells were classified as node-negative. Patients were categorized into 3 classes according to the number of metastatic axillary nodes (N0, N+(1-2), N+(≥3)), which reflected the essential cut-offs for axillary surgery according to the ACOSOG Z0011 criteria.

Statistical analysis

The differences in continuous variables were analyzed using the Kruskal-Wallis test, whereas the differences in the categorical variables were assessed using the Pearson χ2 test or Fisher exact test. Multivariate logistic regression analysis was performed to identify independent factors associated with lymph node status among the variables showing statistical significance on univariate analysis (P < 0.05). Results were provided as odd ratios with 95% confidence intervals. The regression coefficients estimated from the multivariable logistic regression models were illustrated graphically in nomograms, which provided a straightforward method of prediction of the extent of axillary disease. The receiver operating characteristic (ROC) curve was drawn, and the area under the curve (AUC) was used to assess the predictive accuracy of the model. Moreover, bootstrap resampling with 1000 replicates was used to estimate the accuracy of the prediction models. The bias-corrected AUC was calculated as the average AUC over the predictions from the 1000-replicate bootstrap data set applied to the original data set. Goodness-of-fit of the models was assessed using the Hosmer-Lemeshow (HL) test. The prediction models were validated by assessing the calibration curves and bias-corrected concordance index (C-index). Statistical analysis was performed with software (SPSS® software version 23.0, SPSS, Chicago, IL, USA; R 3.5.1, Vienna, Austria; http://www.R-project.org/). The R packages used in this study are as follows: “rms” and “pROC”. Comparisons between 2 areas under the ROC curve were performed using the DeLong method implemented in the pROC package with R. The conventional two-sided tests, and a significance level of 0.05 were used in all analyses.

Results

Clinicopathological characteristics

Patients were categorized into 3 groups based on their ALN status, and their detailed descriptive clinicopathological characteristics are provided in Supplementary Table S1. Patient age, histological grade, vascular invasion, PR status, HER2 status, molecular subtype, clinical T stage, MRI features (tumor size; presence of unifocal, multifocal, and multicentric diseases; tumor location; mass margin; LN LD, SD, and LD/SD; LN margin; hilum; and symmetry), and AUS descriptors (presence or absence of cortical thickening, medulla, and hilum) varied significantly across the 3 different ALN statuses (Table 1,2). No significant differences were observed with regard to the histological types, ER status, Ki67 levels, FGT, BPE, mass/non-mass, mass shape, internal enhancement characteristics of mass/non-mass, distribution of non-mass enhancement (NME), TIC types, and node margin on AUS among the 3 ALN statuses.

 Table 1 

Clinicopathological characteristics of the study sample

Axillary nodal status
N0 (n=200)N+(1-2) (n=119)N+(≥3) (n=78)P
Age0.035
<40y9135(17.5)34(28.6)22(28.2)
≥40y306165(82.5)85(71.4)56(71.8)
Nuclear grade0.002
Low to intermediate264140(73.7)86(74.8)38(52.8)
High10350(26.3)29(25.2)34(47.2)
Unknown201046
Vascular invasion<0.001
No12874(73.3)36(45.0)18(32.1)
Yes10927(26.7)44(55.0)38(67.9)
Missing160993922
PR0.029
Negative10966(33.0)29(24.6)14(17.9)
Positive287134(67.0)89(75.4)64(82.1)
Missing1010
HER2<0.001
-16988(44.0)49(41.2)32(41.0)
1+12333(16.5)51(42.9)39(50.0)
2+5440(20.0)13(10.9)1(1.3)
3+5139(19.5)6(5.0)6(7.7)
Molecular subtype0.002
Luminal313149(76.4)97(82.9)67(85.9)
HER2-positive1717(8.7)00
Basel-like6029(14.9)20(17.1)11(14.1)
Uncertain7510
Clinical T stage0.012
T1186103(51.5)58(48.7)25(32.1)
T221197(48.5)61(51.3)53(67.9)

Note. Except where indicated, data represent number of patients, with percentages in parentheses. N0, lymph node-negative; N+, any lymph node metastasis; N+(1-2), lymph node metastasis involving 1 or 2 nodes; N+(≥ 3), lymph node metastasis involving at least 3 nodes; ER, estrogen receptor; PR, progesterone receptor; HER2, human epidermal growth factor receptor 2.

† St. Gallen surrogate molecular subtype

Univariate and multivariate analyses of nodal status

On univariate analyses, several factors were significantly associated with the extent of axillary node involvement, including age, histological grade, vascular invasion, ER status, PR status, HER2 status, Ki67 level, clinical T stage, MRI parameters (presence of unifocal, multifocal, and multicentric diseases; tumor size; mass margin; tumor location; ALN LD, SD, and LD/SD; node margin; hilum; and symmetry), and AUS evaluation findings (LN hilum, cortex, and medulla) (Table 3). The association with 3 outcomes, including axillary node negativity (N0 vs. any nodal metastasis (N+)), low burden of metastatic axillary disease (N+(1-2) vs. N0), or heavy burden of metastatic axillary disease (N+(≥3) vs. N+(1-2) and N0), were studied via multivariable logistic regression analysis (Table 4). The predictors identified included age, vascular invasion, HER2 status, mass margin, ALN margin, hilum and symmetry on MRI, and the hilum and cortex on US.

 Table 2 

Routine and dynamic contrast-enhanced MRI and axillary ultrasound findings of the study sample

Axillary nodal status
N0 (n=200)N+(1-2) (n=119)N+(≥3) (n=78)P
Tumor size (mm)*23.2(16, 28)22.3(15, 27)22.5(16.5, 26)26.9(20, 32)<0.001
Uni- and multifocal tumors0.016
Unifocal296152(86.4)91(85.8)53(79.1)
Multifocal3217(9.7)11(10.4)4(6.0)
Multicentric217(4.0)4(3.8)10(14.9)
Missing48241311
Tumor location0.003
UIQ5739(19.5)11(9.2)7(9.0)
LIQ2416(8.0)3(2.5)5(6.4)
UOQ13458(29.0)52(43.7)24(30.8)
LOQ3515(7.5)8(6.7)12(15.4)
U5326(13.0)19(16.0)8(10.3)
L1511(5.5)3(2.5)1(1.3)
I105(2.5)5(4.2)0
O3315(7.5)10(8.4)8(10.3)
Other3615(7.5)8(6.7)13(16.7)
Mass margin0.010
Circumscribed128(4.6)2(1.9)2(3.0)
Irregular16395(54.9)46(43.4)22(32.8)
Spiculated17170(40.5)58(54.7)43(64.2)
LN LD (mm)*12.13(11.61,12.71)11.40(10.78,12.07)11.84(10.97,12.80)14.42(13.05,15.88)<0.001
LN SD (mm)*7.57(7.22,7.89)6.69(6.35,7.03)7.71(7.06,8.41)9.58(8.70,10.41)<0.001
LN LD/SD1.66(1.61,1.71)1.74(1.66,1.83)1.60(1.51,1.70)1.53(1.46,1.61)0.004
LN margin on MRI<0.001
Smooth322184(96.3)94(81.0)44(58.7)
Irregular607(3.7)22(19.0)31(41.3)
Missing15933
LN hilum on MRI<0.001
Present222140(72.9)59(50.4)23(30.3)
Absent16352(27.1)58(49.6)53(69.7)
Missing12822
LN symmetry on MRI<0.001
Yes256166(86.5)70(60.3)20(26.7)
No12726(13.5)46(39.7)55(73.3)
Missing14833
Absent medulla on US<0.001
No292191(95.5)66(82.5)40(69.0)
Yes439(4.5)14(17.5)18(31.0)
Missing5903920
ALN cortical thickening on US<0.001
N0285184(92.0)62(78.5)39(69.6)
Yes5016(8.0)17(21.5)17(30.4)
Missing6204022
ALN hilum on US<0.001
Present295188(94.0)69(86.3)38(64.4)
Absent4412(6.0)11(13.7)21(35.6)
Missing5803919

Note. Except where indicated, data represent the number of patients, with percentages in parentheses. N0, lymph node-negative; N+, any lymph node metastasis; N+(1-2), lymph node metastasis involving 1 or 2 nodes; N+(≥3), lymph node metastasis involving at least 3 nodes; LN, lymph node; UIQ, upper inner quadrant; UOQ, upper outer quadrant; LOQ, lower outer quadrant; LIQ, lower inner quadrant; U, upper quadrant; L, lower quadrant; I, inner quadrant; O, outer quadrant; LD, long diameter; SD, short diameter; ALN, axillary lymph nodes; US, ultrasonography; MRI, magnetic resonance imaging.

* Data are medians, with interquartile range in parentheses.

† Other regions included the central, subareolar area.

 Table 3 

Univariable regression analyses between possible predictor variables and axillary nodal status

Axillary nodal status
N0 vs. N+ (n=397)N+(1-2) vs. N0 (n=319)N+(≥3) vs. N0 and N+(1-2) (n=397)
Odds ratioPOdds ratioPOdds ratioP
Age0.0030.0210.366
<40y111
≥40y2.015(1.278,3.178)0.530(0.309,0.910)0.778(0.453,1.339)
Histological grade0.1190.832<0.001
I-II111
III0.703(0.451,1.095)0.944(0.555,1.605)2.560(1.508,4.344)
Vascular invasion<0.001<0.001<0.001
No111
Yes0.219(0.133,0.359)3.759(2.168,6.518)3.318(1.98,5.562)
ER0.0580.3640.032
Negative111
Positive0.634(0.396,1.015)1.278(0.753,2.172)2.116(1.066,4.200)
PR0.0140.1140.037
Negative111
Positive0.571(0.364,0.894)1.512(0.906,2.523)1.947(1.041,3.643)
HER2<0.001<0.0010.001
-111
1+0.338(0.205,0.557)2.776(1.585,4.860)1.988(1.158,3.413)
2+2.630(1.333,5.188)0.584(0.285,1.195)0.081(0.011,0.606)
3+2.991(1.465,6.109)0.276(0.109,0.699)0.571(0.224,1.454)
Ki670.2420.9350.030
Low (<14%)111
High0.774(0.505,1.188)1.020(0.630,1.654)1.937(1.067,3.520)
Clinical T stage0.0620.6330.004
T1111
T20.686(0.461,1.019)1.117(0.709,1.758)2.160(1.280,3.647)
Uni- and multifocal tumors0.2800.9790.005
Unifocal111
Multifocal1.074(0.517,2.230)1.081(0.485,2.409)0.655(0.22,1.946)
Multicentric0.474(0.186,1.207)0.954(0.272,3.350)4.168(1.684,10.318)
Tumor size (mm)*0.659(0.436,0.994)0.0470.981(0.601,1.601)0.942.834(1.708,4.702)<0.001
Mass margin0.0040.0440.030
Circumscribed111
Irregular0.699(0.202,2.414)1.937(0.395,9.488)0.780(0.16,3.8)
Spiculated0.347(0.100,1.195)3.371(0.689,16.495)1.667(0.351,7.907)
LN LD (mm)*0.623(0.400,0.971)0.0361.122(0.668,1.885)0.6632.639(1.529,4.53)<0.001
LN SD (mm)*0.424(0.278,0.645)01.563(0.963,2.538)0.0713.807(2.226,6.511)<0.001
LN LD/SD1.556(1.028,2.353)0.0360.651(0.403,1.052)0.790.737(0.434,1.253)0.260
LN margin on MRI<0.001<0.001<0.001
Smooth111
Irregular0.099(0.044,0.225)6.152(2.536,14.922)6.754(3.714,12.281)
LN hilum on MRI<0.001<0.001<0.001
Present111
Absent0.274(0.179,0.421)2.647(1.634,4.287)4.169(2.425,7.168)
LN symmetry on MRI<0.001<0.001<0.001
Yes111
No0.140(0.085,0.230)4.196(2.406,7.317)9.014(5.068,16.032)
Tumor location0.0270.0520.060
UIQ111
LIQ0.923(0.334,2.550)0.665(0.163,2.704)1.880(0.531,6.649)
UOQ0.352(0.183,0.678)3.179(1.477,6.843)1.558(0.630,3.856)
LOQ0.346(0.145,0.828)1.891(0.637,5.614)3.727(1.298,10.702)
U0.444(0.205,0.966)2.591(1.061,6.329)1.270(0.426,3.782)
L1.269(0.355,4.535)0.967(0.229,4.087)0.510(0.058,4.502)
I0.462(0.119,1.798)3.545(0.867,14.502)0
O0.385(0.159,0.931)2.364(0.833,6.708)2.286(0.744,7.021)
Other†0.330(0.139,0.784)1.891(0.637,5.614)4.037(1.423,11.458)
Absent medulla on US<0.001 0.0100.001
No111
Yes0.156(0.072,0.339)4.502(1.862,10.885)5.028(2.494,10.136)
ALN cortical thickening on US<0.001 0.020<0.001
No111
Yes0.258(0.136,0.491)3.153(1.503,6.615)3.249(1.653,6.386)
ALN hilum on US<0.001 0.038<0.001
Present111
Absent0.213(0.105,0.432)2.498(1.053,5.922)6.175(3.120,12.221)

Note. Values in parentheses are 95% confidence intervals. N0, lymph node-negative; N+, any lymph node metastasis; N+(1-2), lymph node metastasis involving 1 or 2 nodes; N+(≥3), lymph node metastasis involving at least 3 nodes; ER, estrogen receptor; PR, progesterone receptor; HER2, human epidermal growth factor receptor 2; Ki67, antigen identified by monoclonal antibody Ki-67; UIQ, upper inner quadrant; UOQ, upper outer quadrant; LOQ, lower outer quadrant; LIQ, lower inner quadrant; U, upper quadrant; L, lower quadrant; I, inner quadrant; O, outer quadrant; LD, long diameter; SD, short diameter; LN, lymph node; ALN, axillary lymph nodes; US, ultrasonography; MRI, magnetic resonance imaging.

* Data are medians, with interquartile range in parentheses.

† Other regions included the central, subareolar area.

Bold values denote p < 0.05

 Table 4 

Multivariable regression analyses for the prediction of axillary status

Axillary nodal status
N0 vs. N+ (n=397)N+(1-2) vs. N0 (n=319)N+(≥3) vs. N0+ and N+(1-2) (n=397)
Odds ratioPOdds ratioPOdds ratiop
Age (years)0.018
<40 years1
≥40 years2.69(1.184,6.115)
Vascular invasion<0.001<0.0010.014
No111
Yes0.14(0.064,0.308)7.323(3.189,16.814)3.35(1.276,8.791)
HER20.0070.005
-11
1+0.34(0.153,0.753)3.594(1.544,8.368)
2+1.772(0.625,5.023)0.923(0.325,2.625)
3+1.425(0.462,4.395)0.538(0.152,1.900)
Mass margin on MRI0.0100.028
Circumscribed11
Irregular0.91(0.125,6.632)1.599(0.175,14.613)
Spiculated0.315(0.044,2.259)4.113(0.460,36.767)
LN margin on MRI0.0070.006
Smooth11
Irregular0.17(0.047,0.609)4.719(1.547,14.399)
LN symmetry on MRI0.0010.0230.002
Yes111
No0.258(0.114,0.585)2.882(1.159,7.165)4.66(1.729,12.555)
LN hilum on MRI0.010
Present1
Absent2.903(1.288,6.543)
ALN cortical thickening on US0.017
No1
Yes0.297(0.109,0.808)
ALN hilum on US0.004
Present1
Absent5.517(1.736,17.534)

Note. Values in parentheses are 95% confidence intervals. N0, lymph node-negative; N+, any lymph node metastasis; N+(1-2), lymph node metastasis involving 1 or 2 nodes; N+(≥3), lymph node metastasis involving at least 3 nodes; HER2, human epidermal growth factor receptor 2; LN, lymph node; ALN, axillary lymph node; US, ultrasonography; MRI, magnetic resonance imaging.

The absence/presence of vascular invasion and symmetry/asymmetry of the ALN on MRI were strongly associated with ALN negativity as well as a low and heavy burden of axillary disease. Vascular invasion and asymmetry of ALN was negatively associated with N0 disease. Increasing age, HER2-positive status, lack of cortical thickening on AUS, and circumscribed mass margin and smooth LN margin on MRI were all associated with ALN negativity. The factors associated with axillary metastatic disease involving 1 or 2 lymph nodes are as follows: vascular invasion, HER2-negative status, spiculated mass margin, asymmetry, and the absence of hilum on axillary MRI. Vascular invasion, irregular margin and asymmetry of the ALN on MRI, and absence of the LN hilum on US were found to be positively associated with heavy burden of axillary involvement strongly.

Nomograms predicting axillary nodal status

The preoperative imaging parameters identified from the multivariable regression analyses were used to construct 3 nomograms that predicted the extent of axillary nodal disease. A score proportional to the log of the OR (estimated regression coefficient) was assigned to each independent predictor (Fig. 3A,B,C).

Receiver operating characteristic curve analysis and internal validation

The discriminatory ability of the nomograms for predicting each of the 3 axillary nodal status groups was investigated using ROC curves (Fig. 4A,B,C). The AUC for a disease-free axilla was 0.809 (95% CI, 0.756-0.863). Using 1000 resampled bootstrap data sets, the bias-corrected AUC for N0 was 0.773, with a decrease (-0.036) in discriminative ability. The nomogram displayed a bias-corrected C-index of 0.793. The Goodness-of-Fit analysis with the HL test resulted in a P value of 0.906, which indicated that the model fit well. The AUC for 1 or 2 positive nodes was 0.749 (95% CI, 0.675-0.823), with a bias-corrected AUC of 0.696 (-0.053), whereas that for ≥3 metastatic nodes was 0.874 (95% CI, 0.819-0.929), with a bias-corrected AUC of 0.851 (-0.023). For the latter 2 nomograms, the bias-corrected C-index was 0.723 and 0.853, respectively, whereas the P value on the HL test for goodness-of-fit was 0.815 and 0.295, respectively. The calibration was also assessed graphically (Fig. 4D,E,F). Moreover, we assessed the additional contributions made by clinical parameters. The predictive ability of the models that considered both preoperative imaging parameters and pathological variables (including vascular invasion and HER2 status) to determine a disease-free axilla (AUC, 0.871; 95% CI, 0.827-0.916; bias-corrected C-index, 0.854), 1 or 2 positive nodes (AUC, 0.821; 95% CI, 0.756-0.885; bias-corrected C-index, 0.791), and ≥3 metastatic nodes (AUC, 0.898; 95% CI, 0.851-0.946; bias-corrected C-index, 0.871) was superior to that of the above constructed models. For a diseased axilla and low burden of LN metastasis, the performance of the models were significantly improved after adding pathological factors (P < 0.05); this improvement, however, was not significant when the heavy burden of lymph node metastasis was considered (P > 0.05; Fig. 4A,B,C).

Discussion

In the present study, patient age and 6 imaging parameters were included in the final models. Three nomograms were developed and showed good performance for the evaluation of ALN status via ROC curve analysis; internal validation demonstrated good discrimination. The nomograms were able to identify a negative axilla, as well as low- and high-risk groups based on the extent of ALN metastatic involvement, and can therefore help clinicians to determine appropriate axillary treatment options—no axillary staging in patients with N0, sentinel node staging in patients with N+(1-2), and completion ALND or neoadjuvant therapy in patients with N+(≥3). In addition, we provide evidence of the independent relationship between preoperative MRI and US parameters and clinical variables, and the lymphatic spread of breast cancer.

 Figure 3 

(A) Nomogram predicting disease-free axilla (N0) vs. any nodal metastasis (N+), (B) Nomogram predicting low burden of axillary diseases involving 1 or 2 positive nodes (N+(1-2)) vs. disease-free axilla (N0) and (C) Nomogram predicting heavy burden of axillary diseases involving at least 3 positive nodes (N≥3) vs. disease-free axilla (N0) and 1 or 2 positive nodes (N+(1-2)), considering age; mass margin; ALN hilum, margin and symmetry on MRI; ALN cortical thickening and hilum on US. ALN, axillary lymph node; MRI, magnetic resonance imaging; US, ultrasonography.

J Cancer Image
 Figure 4 

Calibration plots of (A) the predictive model predicting disease-free axilla (N0) vs. any nodal metastasis (N+), (B) the predictive model predicting low burden of axillary disease involving 1 or 2 nodes (N+(1-2)) vs. disease-free axilla (N0) and (C) the predictive model predicting heavy burden of axillary disease involving at least 3 nodes (N≥3) vs. disease-free axilla (N0) and 1 or 2 positive lymph nodes (N+(1-2)). Receiver operating characteristics (ROC) curves representing the discriminatory ability of the nomogram in (D) predicting a negative axilla, (E) axillary disease involving 1 or 2 positive nodes and (F) axillary disease involving at least three positive nodes. The red plot shows the ROC curve of preoperative parameters and pathological variables, whereas the blue plot shows the ROC curve of the model considering preoperative parameters. Preoperative parameters include age, mass margin, ALN hilum, margin and symmetry on MRI, ALN cortical thickening and hilum on US; the pathological variables include vascular invasion and human epidermal growth factor receptor 2 (HER2) status. ALN, axillary lymph node; MRI, magnetic resonance imaging; US, ultrasonography

J Cancer Image

On multivariate analyses, 9 variables emerged as independent predictors for ALN status, including age, vascular invasion, HER2 status, mass margin, ALN features (margin, hilum, and symmetry) on MRI, and AUS evaluation of the LN hilum and cortex. Age (<40 years), vascular invasion, and absence of ALN hilum and cortical thickening on US are reportedly risk factors for ALN metastasis, in accordance with most previous publications (10,25-29). However, the prediction value of HER2 status in previous studies was uncertain, with some studies showing no predictive value (25,30). In the present study, we found that HER2-negative status was associated with a higher burden of ALN metastasis. This finding may seem counterintuitive (31), but is similar to the finding of Fei et al (32); its inclusion in the present study should be corroborated.

We used a mix of quantitative and qualitative descriptors that are common interpretation criteria frequently used in assessment of the ALN via MRI (17). The disappearance of the hilum, irregular margin and asymmetry on axillary MRI are independent predictors of the extent of node metastasis. According to previous reports, the absence of LN hilum was associated with lymph node metastasis (27,33,34). Fusco et al showed that the accuracy of ALN smoothness for discriminating metastatic and non-metastatic lymph nodes was 77.2%. Previous MRI prediction models rarely used the descriptor “asymmetry” for assessing nodal status. Pascal et al demonstrated that “asymmetry” and “irregular margin” were significant predictors for ALN metastasis by using chi-squared automatic interaction detection (17). The results of our study was consistent with these findings.

To our knowledge, only a few studies have sought to develop a predictive model of ALN status that incorporates routine and dynamic contrast MRI parameters of primary lesions based on the 2013 BI-RADS lexicon. The mass margin could serve as an independent prognostic factor for ALN negativity and 1 or 2 positive lymph nodes in this study. Previous studies found that lesion segmentation and extraction signatures or BI-RADS features of the primary mass could potentially predict ALN metastasis or breast cancer recurrence risk (35-37). Woodard et al found that irregular mass margins on MRI may be indicative of a more aggressive malignancy (35). Nevertheless, the relationship between mass morphological signatures and axillary node metastasis needs more researches to be verified.

In previous studies, models have been developed to predict the probability of ALN staging in patients with EBC, with AUC ranging from 0.702 to 0.864 (10,24,26,28,31,32,38). When compared with those models, our models showed some advantages. First, our model contains 7 variables that were available prior to surgery and showed good discrimination ability, yielding an AUC of 0.749-0.874. Given that vascular invasion and HER2 status can be obtained after core needle biopsy of the primary tumor, the classification performance of our models can be improved significantly. Nevertheless, information on vascular invasion and HER2 status was not often available preoperatively and was hence not included in the nomograms. However, predictive factors such as pathological tumor size and multifocality in previous models are only generally available postoperatively, which may limit their clinical application. Secondly, we developed 3 nomograms using comprehensive data to more accurately evaluate the extent of lymph node involvement. Physicians can better inform patients and can establish a more appropriate treatment protocol based on the risk of ALN metastasis with our user-friendly calculators.

Nevertheless, the study still has some limitations due to its retrospective nature. Since the study has a single-center cohort, inevitable selection bias in the inclusion and exclusion criteria could have influenced the results. Although internal validation was performed by bootstrapping, external validation is still needed to evaluate its predictive ability and generalizability. Moreover, the risk factors may differ when measured by different interpreters due to interobserver variability. Hence, the nomogram results should be interpreted with caution. Radiomics and machine learning approaches could play a better role in stratifying the risk in future studies. In addition, patients with negative SLNB findings were considered to have “negative axillary disease”, even though they did not undergo ALND and thus did not have definite pathologic results. This inherent limitation is unavoidable because patients with negative SLNB are required to forgo ALND in clinical practices. Finally, the study cohort is extracted from the Chinese population. As breast cancer patients in China are usually younger and the breast FGT of Chinese women is different from those of women from other countries, our prediction model cannot be directly applied to patients from other non-Asian countries.

In conclusion, the value of the 3 nomograms as preoperative guidance tools will depend on the preoperative imaging modalities used and routine clinicopathological factors; the methodological limitations, imaging interpretation variability, and population discrepancies should be considered when validating its feasibility in clinical practice.

Supplementary Material

Supplementary table.

Attachment

Acknowledgements

This work was funded by the Science and Technology Planning Project of Guangdong Province, China (No.2016B090918066) and the Science and Technology Program of Guangzhou, China (No.201704020060, 201807010057).

Competing Interests

The authors have declared that no competing interest exists.

References

1. Veronesi U, Paganelli G, Viale G. et al. A randomized comparison of sentinel-node biopsy with routine axillary dissection in breast cancer. N Engl J Med. 2005;349(6):546-553

2. Lyman G, Giuliano A, Somerfield M. et al. American Society of Clinical Oncology guideline recommendations for sentinel lymph node biopsy in early-stage breast cancer. J Clin Oncol. 2005;23(30):7703-7720

3. Giuliano AE, McCall L, Beitsch P. et al. Locoregional recurrence after sentinel lymph node dissection with or without axillary dissection in patients with sentinel lymph node metastases: the American College of Surgeons Oncology Group Z0011 randomized trial. Ann Surg. 2010;252(3):426-432 discussion 432-423

4. Giuliano AE, Ballman K, McCall L. et al. Locoregional Recurrence After Sentinel Lymph Node Dissection With or Without Axillary Dissection in Patients With Sentinel Lymph Node Metastases: Long-term Follow-up From the American College of Surgeons Oncology Group (Alliance) ACOSOG Z0011 Randomized Trial. Ann Surg. 2016;264(3):413-420

5. Maguire A, Brogi E. Sentinel lymph nodes for breast carcinoma: an update on current practice. Histopathology. 2016;68(1):152-167

6. Fleissig A, Fallowfield LJ, Fallowfield LJ. et al. Post-operative arm morbidity and quality of life. Results of the ALMANAC randomised trial comparing sentinel node biopsy with standard axillary treatment in the management of patients with early breast cancer. Breast Cancer Res Treat. 2006;95(3):279-293

7. Del-Bianco P, Zavagno G, Burelli P. et al. Morbidity comparison of sentinel lymph node biopsy versus conventional axillary lymph node dissection for breast cancer patients: results of the sentinella-GIVOM Italian randomised clinical trial. Eur J Surg Oncol. 2008;34(5):508-513

8. I L, U G, G B. et al. Morbidity of sentinel lymph node biopsy (SLN) alone versus SLN and completion axillary lymph node dissection after breast cancer surgery: a prospective Swiss multicenter study on 659 patients. Ann Surg. 2007;18(4):452-461

9. Kootstra JJ, Hoekstraweebers JE, Rietman JS. et al. A longitudinal comparison of arm morbidity in stage I-II breast cancer patients treated with sentinel lymph node biopsy, sentinel lymph node biopsy followed by completion lymph node dissection, or axillary lymph node dissection. Ann Surg Oncol. 2010;17(9):2384-2394

10. Dihge L, Bendahl PO, Rydén L. Nomograms for preoperative prediction of axillary nodal status in breast cancer. Br J Surg. 2017;104(11):1494-1505

11. Cools-Lartigue J, Meterissian S. Accuracy of Axillary Ultrasound in the Diagnosis of Nodal Metastasis in Invasive Breast Cancer: A Review. World J Surg. 2012;36(1):46-54

12. Ibrahimzada I, Grant CS, Glazebrook KN, Boughey JC. Preoperative axillary ultrasound in breast cancer: safely avoiding frozen section of sentinel lymph nodes in breast-conserving surgery. J Am Coll Surg. 2013;217(1):7-15

13. Lam SW, Jimenez CR, Boven E. Breast cancer classification by proteomic technologies: current state of knowledge. Cancer Treat Rev. 2014;40(1):129-138

14. Diepstraten SC, Sever AR, Buckens CF. et al. Value of preoperative ultrasound-guided axillary lymph node biopsy for preventing completion axillary lymph node dissection in breast cancer: a systematic review and meta-analysis. Ann Surg Oncol. 2014;21(1):51-59

15. Cooper KL, Harnan S, Meng Y. et al. Positron emission tomography (PET) for assessment of axillary lymph node status in early breast cancer: A systematic review and meta-analysis. Eur J Surg Oncol. 2011;37(3):187-198

16. Uematsu T, Sano M, Homma K. In vitro high-resolution helical CT of small axillary lymph nodes in patients with breast cancer: correlation of CT and histology. Ajr Am J Roentgenol. 2001;176(4):1069-1074

17. Baltzer PA, Dietzel M, Burmeister HP. et al. Application of MR mammography beyond local staging: is there a potential to accurately assess axillary lymph nodes? evaluation of an extended protocol in an initial prospective study. Ajr Am J Roentgenol. 2011;196(5):641-647

18. Nasu Y, Shikishima H, Miyasaka Y, Nakakubo Y, Ichinokawa K, Kaneko T. A study of the assessment of axillary lymph nodes before surgery for breast cancer using multidetector-row computed tomography. Surg Today. 2010;40(11):1023-1026

19. Ecanow JS, Abe H, Newstead GM, Ecanow DB, Jeske JM. Axillary staging of breast cancer: what the radiologist should know. Radiographics. 2013;33(6):1589-1612

20. Ningbin Luo MD, Danke Su MD, Guanqiao Jin MD. et al. Apparent diffusion coefficient ratio between axillary lymph node with primary tumor to detect nodal metastasis in breast cancer patients. J Magn Reson Imaging. 2013;38(4):824-828

21. Schipper RJ, Paiman ML, Beets-Tan RG. et al. Diagnostic Performance of Dedicated Axillary T2- and Diffusion-weighted MR Imaging for Nodal Staging in Breast Cancer. Radiology. 2015;275(2):345-355

22. Perez EA, Dueck AC, Mccullough AE. et al. Predictability of adjuvant trastuzumab benefit in N9831 patients using the ASCO/CAP HER2-positivity criteria. J Natl Cancer Inst. 2012;104(2):159-162

23. Hammond ME, Hayes DF, Dowsett M. et al. American Society of Clinical Oncology/College Of American Pathologists guideline recommendations for immunohistochemical testing of estrogen and progesterone receptors in breast cancer. J Clin Oncol. 2010;28(16):2784-2795

24. Kim JY, Seo HB, Park S. et al. Early-stage invasive ductal carcinoma: Association of tumor apparent diffusion coefficient values with axillary lymph node metastasis. Eur J Radiol. 2015;84(11):2137-2143

25. Kim GR, Choi JS, Han BK. et al. Preoperative Axillary US in Early-Stage Breast Cancer: Potential to Prevent Unnecessary Axillary Lymph Node Dissection. Radiology. 2018;288:55-63

26. Kim SH, Shin HJ, Shin KC. et al. Diagnostic Performance of Fused Diffusion-Weighted Imaging Using T1-Weighted Imaging for Axillary Nodal Staging in Patients With Early Breast Cancer. Clin Breast Cancer. 2017;17(2):154-163

27. Arslan G, Altintoprak KM, Yirgin IK, Atasoy MM, Celik L. Diagnostic accuracy of metastatic axillary lymph nodes in breast MRI. Springerplus. 2016;5(1):1-6

28. Qiu SQ, Zeng HC, Zhang F. et al. A nomogram to predict the probability of axillary lymph node metastasis in early breast cancer patients with positive axillary ultrasound. Sci Rep. 2016;6:21196

29. Kim WH, Kim HJ, Lee SM. et al. Preoperative axillary nodal staging with ultrasound and magnetic resonance imaging: predictive values of quantitative and semantic features. Br J Radiol. 2018;91(1092):20180507

30. Zhang J, Li X, Huang R. et al. A nomogram to predict the probability of axillary lymph node metastasis in female patients with breast cancer in China: A nationwide, multicenter, 10-year epidemiological study. Oncotarget. 2017;8(21):35311-35325

31. Canavese G, Bruzzi P, Catturich A. et al. A risk score model predictive of the presence of additional disease in the axilla in early-breast cancer patients with one or two metastatic sentinel lymph nodes. Eur J Surg Oncol. 2014;40(7):835-842

32. Xie F, Yang H, Wang S. et al. A Logistic Regression Model for Predicting Axillary Lymph Node Metastases in Early Breast Carcinoma Patients. Sensors. 2012;12(7):9936-9950

33. Razek AAKA. Assessment of axillary lymph nodes in patients with breast cancer with diffusion-weighted MR imaging in combination with routine and dynamic contrast MR imaging. Breast Cancer. 2016;23(3):525-532

34. Mortellaro VE, Marshall J, Singer L. et al. Magnetic resonance imaging for axillary staging in patients with breast cancer. J Magn Reson Imaging. 2009;30(2):309-312

35. Woodard GA, Ray KM, Joe BN, Price ER. Qualitative Radiogenomics: Association between Oncotype DX Test Recurrence Score and BI-RADS Mammographic and Breast MR Imaging Features. Radiology. 2018;286(1):60-70

36. Bhooshan N, Giger ML, Jansen SA, Li H, Lan L, Newstead GM. Cancerous breast lesions on dynamic contrast-enhanced MR images: computerized characterization for image-based prognostic markers. Radiology. 2010;254(3):680-690

37. Dong Y, Feng Q, Yang W. et al. Preoperative prediction of sentinel lymph node metastasis in breast cancer based on radiomics of T2-weighted fat-suppression and diffusion-weighted MRI. Eur Radiol. 2018;28(2):582-591

38. Xie X, Tan W, Chen B. et al. Preoperative prediction nomogram based on primary tumor miRNAs signature and clinical-related features for axillary lymph node metastasis in early-stage invasive breast cancer. Int J Cancer. 2018;142(9):1901-1910

Author contact

Corresponding address Corresponding authors: Li Li and Yaopan Wu; Email: li2sysu.edu.cn and wuyporg.cn; Department of Medical Imaging, Sun Yat-sen University Cancer Center, State Key Laboratory of Oncology in South China, Guangzhou, People's Republic of China; Tel: +86(020) 87342120; Fax: 0086-020-87343392


Received 2018-12-18
Accepted 2019-1-8
Published 2019-1-29


Citation styles

APA
Li, J., Ma, W., Jiang, X., Cui, C., Wang, H., Chen, J., Nie, R., Wu, Y., Li, L. (2019). Development and Validation of Nomograms Predictive of Axillary Nodal Status to Guide Surgical Decision-Making in Early-Stage Breast Cancer. Journal of Cancer, 10(5), 1263-1274. https://doi.org/10.7150/jca.32386.

ACS
Li, J.; Ma, W.; Jiang, X.; Cui, C.; Wang, H.; Chen, J.; Nie, R.; Wu, Y.; Li, L. Development and Validation of Nomograms Predictive of Axillary Nodal Status to Guide Surgical Decision-Making in Early-Stage Breast Cancer. J. Cancer 2019, 10 (5), 1263-1274. DOI: 10.7150/jca.32386.

NLM
Li J, Ma W, Jiang X, Cui C, Wang H, Chen J, Nie R, Wu Y, Li L. Development and Validation of Nomograms Predictive of Axillary Nodal Status to Guide Surgical Decision-Making in Early-Stage Breast Cancer. J Cancer 2019; 10(5):1263-1274. doi:10.7150/jca.32386. https://www.jcancer.org/v10p1263.htm

CSE
Li J, Ma W, Jiang X, Cui C, Wang H, Chen J, Nie R, Wu Y, Li L. 2019. Development and Validation of Nomograms Predictive of Axillary Nodal Status to Guide Surgical Decision-Making in Early-Stage Breast Cancer. J Cancer. 10(5):1263-1274.

This is an open access article distributed under the terms of the Creative Commons Attribution (CC BY-NC) license (https://creativecommons.org/licenses/by-nc/4.0/). See http://ivyspring.com/terms for full terms and conditions.
Popup Image