Evaluasi Dampak SMOTE-NC Terhadap Stabilitas Interpretasi SHAP pada Model XGBoost Teroptimasi untuk Klasifikasi Churn Pelanggan

Haeruzzikri; Robert Marco

doi:10.24252/instek.v11i2.69816

Authors

Haeruzzikri Universitas Amikom Yogyakarta
Robert Marco Universitas Amikom Yogyakarta

DOI:

https://doi.org/10.24252/instek.v11i2.69816

Keywords:

Churn Pelanggan, Explainable AI, SHAP, SMOTE-NC, XGBoost

Abstract

Ketidakseimbangan kelas pada dataset telekomunikasi dapat memicu tingginya angka false negative dalam prediksi customer churn. Pendekatan SMOTE-NC mampu mengatasi masalah tersebut, namun berisiko mendistorsi interpretasi pada model black-box seperti XGBoost. Penelitian ini bertujuan mengevaluasi efektivitas SMOTE-NC dalam meningkatkan deteksi kelas minoritas, sekaligus mengukur stabilitas interpretasi Explainable AI (XAI) pada XGBoost yang dioptimasi dengan Optuna melalui evaluasi multi-seed dan skenario kontrol (ablation study). Ekstraksi fitur dievaluasi dengan TreeSHAP, sementara stabilitas kuantitatif diukur melalui Jaccard Index dan Spearman Rank Correlation pada variasi ambang batas k = 5, 10, 15. Hasil empiris menunjukkan SMOTE-NC secara konsisten meningkatkan kepekaan deteksi (Recall) sebesar +8.55%, dengan kompromi penurunan akurasi global sebesar -2.38%. Analisis XAI mengonfirmasi bahwa kendati terjadi distorsi interpretasi pada representasi fitur kategorikal, komposisi fitur utama (k=10) secara umum berhasil dipertahankan secara konsisten (rerata Jaccard Index 0.7939 ± 0.1233) dengan korelasi hierarki yang sangat kuat (Spearman Correlation 0.8526). Pengujian kontrol membuktikan bahwa optimasi hyperparameter Optuna berfungsi memitigasi degradasi stabilitas eksplanasi yang dipicu oleh SMOTE-NC. Penelitian ini menyimpulkan bahwa kompromi performa akibat interpolasi data sintetis layak secara operasional untuk strategi retensi pelanggan, dengan prioritas intervensi bisnis pada fitur jenis kontrak, masa berlangganan, dan metode pembayaran.

Downloads

Download data is not yet available.

References

[1] H. Ribeiro, B. Barbosa, A. C. Moreira, and R. G. Rodrigues, “Determinants of churn in telecommunication services: a systematic literature review,” Management Review Quarterly, vol. 74, no. 3, pp. 1327–1364, Sep. 2024, doi: 10.1007/s11301-023-00335-7.

[2] A. Dhini and M. Fauzan, “Predicting Customer Churn using ensemble learning: Case Study of a Fixed Broadband Company,” International Journal of Technology, vol. 12, no. 5, pp. 1030–1037, 2021, doi: 10.14716/ijtech.v12i5.5223.

[3] Ibrahim Adedeji Adeniran, Christianah Pelumi Efunniyi, Olajide Soji Osundare, and Angela Omozele Abhulimen, “Implementing machine learning techniques for customer retention and churn prediction in telecommunications,” Computer Science & IT Research Journal, vol. 5, no. 8, pp. 2011–2025, Aug. 2024, doi: 10.51594/csitrj.v5i8.1489.

[4] R. Suguna, J. Suriya Prakash, H. Aditya Pai, T. R. Mahesh, V. Vinoth Kumar, and T. E. Yimer, “Mitigating class imbalance in churn prediction with ensemble methods and SMOTE,” Sci. Rep., vol. 15, no. 1, p. 16256, May 2025, doi: 10.1038/s41598-025-01031-0.

[5] R. Rofik and J. Unjung, “Evaluation of Ridge Classifier and Logistic Regression for Customer Churn Prediction on Imbalanced Telecommunication Data,” Scientific Journal of Informatics, vol. 12, no. 2, pp. 311–326, Jul. 2025, doi: 10.15294/sji.v12i2.24620.

[6] J. Fonseca and F. Bacao, “Geometric SMOTE for imbalanced datasets with nominal and continuous features,” Expert Syst. Appl., vol. 234, p. 121053, Dec. 2023, doi: 10.1016/j.eswa.2023.121053.

[7] N. Ahmad, M. J. Awan, H. Nobanee, A. M. Zain, A. Naseem, and A. Mahmoud, “Customer Personality Analysis for Churn Prediction Using Hybrid Ensemble Models and Class Balancing Techniques,” IEEE Access, vol. 12, pp. 1865–1879, 2024, doi: 10.1109/ACCESS.2023.3334641.

[8] S. Ouf, K. T. Mahmoud, and M. A. Abdel-Fattah, “A proposed hybrid framework to improve the accuracy of customer churn prediction in telecom industry,” J. Big Data, vol. 11, no. 1, Dec. 2024, doi: 10.1186/s40537-024-00922-9.

[9] S. K. Wagh, A. A. Andhale, K. S. Wagh, J. R. Pansare, S. P. Ambadekar, and S. H. Gawande, “Customer churn prediction in telecom sector using machine learning techniques,” Results in Control and Optimization, vol. 14, Mar. 2024, doi: 10.1016/j.rico.2023.100342.

[10] R. Krishna, D. Jayanthi, D. S. Shylu Sam, K. Kavitha, N. K. Maurya, and T. Benil, “Application of machine learning techniques for churn prediction in the telecom business,” Results in Engineering, vol. 24, Dec. 2024, doi: 10.1016/j.rineng.2024.103165.

[11] S. M. Shrestha and A. Shakya, “A Customer Churn Prediction Model using XGBoost for the Telecommunication Industry in Nepal,” in Procedia Computer Science, Elsevier B.V., 2022, pp. 652–661. doi: 10.1016/j.procs.2022.12.067.

[12] V. Hassija et al., “Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence,” Cognit. Comput., vol. 16, no. 1, pp. 45–74, Jan. 2024, doi: 10.1007/s12559-023-10179-8.

[13] M. Imani and H. R. Arabnia, “Hyperparameter Optimization and Combined Data Sampling Techniques in Machine Learning for Customer Churn Prediction: A Comparative Analysis,” Technologies (Basel)., vol. 11, no. 6, p. 167, Nov. 2023, doi: 10.3390/technologies11060167.

[14] Andy Hermawan, Nila Rusiardi Jayanti, Zia Tabaruk, Faizal Lutfi Yoga Triadi, Aji Saputra, and M.Rahmat Hidayat Syachrudin, “Membangun Model Prediksi Churn Pelanggan yang Akurat (Studi Kasus tentang TELCO Company),” Merkurius : Jurnal Riset Sistem Informasi dan Teknik Informatika, vol. 2, no. 6, pp. 67–81, Oct. 2024, doi: 10.61132/merkurius.v2i6.398.

[15] D. Asif, M. S. Arif, and A. Mukheimer, “A data-driven approach with explainable artificial intelligence for customer churn prediction in the telecommunications industry,” Results in Engineering, vol. 26, Jun. 2025, doi: 10.1016/j.rineng.2025.104629.

[16] A. El Attar and M. El-Hajj, “Explainable AI-driven customer churn prediction: a multi-model ensemble approach with SHAP-based feature analysis,” Front. Artif. Intell., vol. 9, Feb. 2026, doi: 10.3389/frai.2026.1748799.

[17] M. Alkhozae, “An Explainable Credit Card Fraud Detection Model using Machine Learning and Deep Learning Approaches,” Journal of Applied Data Sciences, vol. 6, no. 4, pp. 2838–2857, Dec. 2025, doi: 10.47738/jads.v6i4.962.

[18] S. Mammadov, I. Abdulrashid, and J.-B. Sheu, “Managing Trust in AI Systems: Evidence From Controlled Evaluation of Explanation Stability Under Data Imbalance,” IEEE Trans. Eng. Manag., vol. 73, pp. 3083–3096, 2026, doi: 10.1109/TEM.2026.3687235.

[19] M. KIM, “Does Class Imbalance Undermine SHAP Explanations? A Multi-Domain Empirical Study of Feature Attribution Stability,” Research Square [Preprint], Apr. 2026, doi: 10.21203/rs.3.rs-9489204/v1.

[20] BlastChar, “Dataset Teleco Customer Churn IBM,” Kaggle.com. Accessed: Jun. 30, 2026. [Online]. Available: https://www.kaggle.com/datasets/blastchar/telco-customer-churn

[21] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama, “Optuna: A Next-generation Hyperparameter Optimization Framework,” Preprint, Aug. 2019.

[22] V. Sinap, “Bankruptcy Prediction with Optuna-Enhanced Ensemble Machine Learning Methods: A Comparison of Oversampling and Undersampling Techniques,” DÜMF Mühendislik Dergisi, vol. 16, no. 1, pp. 97–113, Mar. 2025, doi: 10.24012/dumf.1597564.

[23] S. Wu, W.-C. Yau, T.-S. Ong, and S.-C. Chong, “Integrated Churn Prediction and Customer Segmentation Framework for Telco Business,” IEEE Access, vol. 9, pp. 62118–62136, 2021, doi: 10.1109/ACCESS.2021.3073776.

[24] V. Verma and R. K. Aggarwal, “A comparative analysis of similarity measures akin to the Jaccard index in collaborative recommendations: empirical and theoretical perspective,” Soc. Netw. Anal. Min., vol. 10, no. 1, p. 43, Dec. 2020, doi: 10.1007/s13278-020-00660-9.

[25] P. Schober, C. Boer, and L. A. Schwarte, “Correlation Coefficients: Appropriate Use and Interpretation,” Anesth. Analg., vol. 126, no. 5, pp. 1763–1768, May 2018, doi: 10.1213/ANE.0000000000002864.

[26] W. Pirie, “Spearman Rank Correlation Coefficient,” in Encyclopedia of Statistical Sciences, Wiley, 2005. doi: 10.1002/0471667196.ess2499.pub2.