ANALISIS TRADE-OFF ANTARA PERFORMA DAN STABILITAS PADA KLASIFIKASI SENTIMEN WISATA BERBAHASA INDONESIA DENGAN TEKNIK DATA BALANCING

Nurul Chamidah; Intan Hesti Indriana; Bambang Triwahyono; Didit Widiyanto; Indra Permana Solihin; Rio Wirawan

doi:10.24252/instek.v11i1.66678

Authors

Nurul Chamidah Fakultas Ilmu Komputer, Universitas Pembangunan Nasional "Veteran" Jakarta
Intan Hesti Indriana Fakultas Ilmu Komputer, Universitas Pembangunan Nasional ”Veteran” Jakarta
Bambang Triwahyono Fakultas Ilmu Komputer, Universitas Pembangunan Nasional ”Veteran” Jakarta
Didit Widiyanto Fakultas Ilmu Komputer, Universitas Pembangunan Nasional ”Veteran” Jakarta
Indra Permana Solihin Fakultas Ilmu Komputer, Universitas Pembangunan Nasional ”Veteran” Jakarta
Rio Wirawan Fakultas Ilmu Komputer, Universitas Pembangunan Nasional ”Veteran” Jakarta

DOI:

https://doi.org/10.24252/instek.v11i1.66678

Keywords:

dataset kecil, ketidakseimbangan data, klasifikasi sentimen, SMOTE, stabilitas

Abstract

Analisis sentimen pada dataset berukuran kecil sering menghadapi masalah ketidakseimbangan kelas yang dapat menurunkan performa klasifikasi. Berbagai teknik penanganan seperti Synthetic Minority Oversampling Technique (SMOTE) dan Random Under Sampling (RUS) telah digunakan untuk meningkatkan performa, tapi aspek stabilitas model masih relatif jarang diperhatikan. Penelitian ini bertujuan untuk menganalisis performa dan stabilitas model klasifikasi sentimen pada dataset ulasan wisata berbahasa Indonesia yang berukuran kecil dan tidak seimbang. Selain itu, penelitian ini mengintegrasikan metrik stabilitas, yaitu Stability Sensitivity Index (SSI) dan Relative Robustness Score (RRS), yang masih jarang dieksplorasi dalam klasifikasi sentimen berbahasa Indonesia. Model yang digunakan adalah Logistic Regression dengan representasi fitur TF-IDF. Eksperimen dilakukan menggunakan tiga skenario, yaitu tanpa balancing, SMOTE, dan RUS, serta dievaluasi menggunakan 5-fold cross-validation. Hasil menunjukkan bahwa SMOTE menghasilkan F1-score tertinggi sebesar 0.926, sedangkan metode RUS memiliki stabilitas terbaik dengan standar deviasi terendah dan nilai RRS tertinggi. Temuan ini menunjukkan adanya trade-off antara performa dan stabilitas, di mana model dengan performa terbaik tidak selalu paling stabil. Penelitian ini memberikan kerangka evaluasi yang lebih komprehensif untuk klasifikasi sentimen pada dataset kecil dan tidak seimbang.

Downloads

Download data is not yet available.

References

[1] L. Zhang, S. Wang, and B. Liu, “Deep learning for sentiment analysis: A survey,” Wiley Interdiscip. Rev. Data Min. Knowl. Discov., vol. 8, no. 4, 2018, doi: 10.1002/widm.1253.

[2] M. K. Chandan and S. Mandal, “A comprehensive survey on sentiment analysis: Framework, techniques, and applications,” Comput. Sci. Rev., vol. 58, p. 100777, Nov. 2025, doi: 10.1016/J.COSREV.2025.100777.

[3] M. Rezapour, “Sentiment classification of skewed shoppers’ reviews using machine learning techniques,” Engineering Reports, 2020.

[4] M. Lango, “Tackling the Problem of Class Imbalance in Multi-class Sentiment Classification,” Foundations of Computing and Decision Sciences, 2019.

[5] W. Chen, K. Yang, Z. Yu, Y. Shi, and C. L. P. Chen, “A survey on imbalanced learning: latest research, applications and future directions,” Artif. Intell. Rev., vol. 57, no. 6, p. 137, 2024, doi: 10.1007/s10462-024-10759-6.

[6] M. M. Khan and M. Alkhathami, “Anomaly detection in IoT-based healthcare: machine learning for enhanced security,” Sci. Rep., vol. 14, no. 1, p. 5872, 2024, doi: 10.1038/s41598-024-56126-x.

[7] J. Park, S. Kwon, and S.-P. Jeong, “A study on improving turnover intention forecasting by solving imbalanced data problems: focusing on SMOTE and generative adversarial networks,” J. Big Data, vol. 10, no. 1, p. 36, 2023, doi: 10.1186/s40537-023-00715-6.

[8] D. Andriyani, A. Faqih, and S. E. Permana, “The Effect of SMOTE Application on SVM Performance in Sentiment Classification,” Journal of Artificial Intelligence and Engineering Applications, 2025.

[9] N. S. Sediatmoko, Y. Nataliani, and I. Suryady, “Sentiment Analysis of Customer Review Using Classification Algorithms and SMOTE,” Indonesian Journal of Information Systems, 2024.

[10] M. Al-Khazaleh, M. Alian, and M. Jaradat, “Sentiment analysis of imbalanced Arabic data using sampling techniques,” Bulletin of Electrical Engineering and Informatics, 2024.

[11] G. Douzas and F. Bacao, “Geometric SMOTE: a geometrically enhanced drop-in replacement for SMOTE,” Inf. Sci. (N. Y)., vol. 501, pp. 118–135, 2019, doi: 10.1016/j.ins.2019.06.007.

[12] A. Saekhu, B. Berlilana, and D. Saputra, “Comparative Analysis of Data Balancing Techniques for Machine Learning Classification,” Jurnal Teknik Informatika, 2025.

[13] X. Bouthillier, C. Laurent, and P. Vincent, “Accounting for variance in machine learning benchmarks,” Proceedings of Machine Learning and Systems, vol. 3, pp. 747–772, 2021, doi: 10.48550/arXiv.2106.04838.

[14] C. Nadeau and Y. Bengio, “Inference for the generalization error,” Mach. Learn., vol. 52, no. 3, pp. 239–281, 2003, doi: 10.1023/A:1024068626366.

[15] H. Do, P. W. C. Prasad, A. Maag, and A. Alsadoon, “Deep learning for aspect-based sentiment analysis: A comparative review,” Expert Syst. Appl., vol. 118, pp. 272–299, 2019, doi: 10.1016/j.eswa.2018.10.003.

[16] N. Chamidah, D. Widiyanto, H. B. Seta, and A. A. Aziz, “The impact of oversampling and undersampling on aspect-based sentiment analysis of Indramayu tourism using logistic regression,” Revue d’Intelligence Artificielle, vol. 38, no. 3, pp. 795–804, 2024, doi: 10.18280/ria.380306.

[17] Ni Made Gita Satviki Nirmala and Ngurah Agus Sanjaya ER, “Analisis Sentimen dengan Logistic Regression untuk Deteksi Kata pada Livin’ by Mandiri,” Jurnal Nasional Teknologi Informasi dan Aplikasinya, vol. 2, no. 4, pp. 869–878, 2024, doi: 10.24843/JNATIA.2024.v02.i04.p25.

[18] H. P. Singh, N. Singh, A. Mishra, S. K. Sen, M. Swarnkar, and D. Pandey, “Logistic Regression based Sentiment Analysis System: Rectify,” in 2024 IEEE International Conference on Big Data & Machine Learning (ICBDML), 2024, pp. 186–191. doi: 10.1109/ICBDML60909.2024.10577296.

[19] C. D. Manning, P. Raghavan, and H. Schutze, Introduction to Information Retrieval. Cambridge University Press, 2008.

[20] Md. S. Islam et al., “‘Challenges and future in deep learning for sentiment analysis: a comprehensive review and a proposed novel hybrid approach,’” Artif. Intell. Rev., vol. 57, no. 3, p. 62, 2024, doi: 10.1007/s10462-023-10651-9.

[21] S. Dixit, W. Mao, K. K. McDade, M. Schäferhoff, O. Ogbuoji, and G. Yamey, “Tracking financing for global common goods for health: A machine learning approach using natural language processing techniques,” Front. Public Health, vol. Volume 10-2022, 2022, [Online]. Available: https://www.frontiersin.org/journals/public-health/articles/10.3389/fpubh.2022.1031147

[22] M. F. Porter, “An algorithm for suffix stripping,” Program, 1980.

[23] A. P. Pamungkas, A. Mahendra, I. Wahid, and F. Aziz, “Comparison of Machine Learning Models for Classifying Consumer Sentiment of Coffee Shops on Social Media X,” Jurnal Teknik Pertanian Lampung (Journal of Agricultural Engineering), vol. 14, no. 5, pp. 1905–1912, Oct. 2025, doi: 10.23960/JTEPL.V14I5.1905-1912.

[24] K. Schouten and F. Frasincar, “Aspect-based sentiment analysis: A survey,” IEEE Trans. Knowl. Data Eng., vol. 28, no. 3, pp. 813–830, 2016, doi: 10.1109/TKDE.2015.2485209.

[25] X. Huang, J. Li, J. Wu, J. Chang, and D. Liu, “Transfer Learning With Document-Level Data Augmentation for Aspect-Level Sentiment Classification,” IEEE Trans. Big Data, vol. 9, no. 6, pp. 1643–1657, 2023, doi: 10.1109/TBDATA.2023.3310267.

[26] R. F. et al. Baumeister, “Bad is stronger than good,” Review of General Psychology, vol. 5, no. 4, pp. 323–370, 2001, doi: 10.1037/1089-2680.5.4.323.

[27] G. Salton and C. Buckley, “Term-weighting approaches in automatic text retrieval,” Inf. Process. Manag., 1988.

[28] D. Hosmer and S. Lemeshow, Applied Logistic Regression. Wiley, 2013.

[29] A. Genkin, D. Lewis, and D. Madigan, “Large-scale Bayesian logistic regression for text categorization,” in Technometrics, 2007.

[30] B. Krawczyk, “Learning from imbalanced data: open challenges and future directions,” Progress in Artificial Intelligence, vol. 5, no. 4, pp. 221–232, 2016, doi: 10.1007/s13748-016-0094-0.

[31] H. He and Y. Ma, “Learning from class-imbalanced data: Review of methods and applications,” Expert Syst. Appl., vol. 73, pp. 220–239, 2017, doi: 10.1016/j.eswa.2016.12.035.

[32] N. V. et al. Chawla, “SMOTE: Synthetic Minority Over-sampling Technique,” Journal of Artificial Intelligence Research, 2002.

[33] A. O. Mostafa and T. M. Ahmed, “Enhanced Emotion Analysis Model using Machine Learning in Saudi Dialect: COVID-19 Vaccination Case Study,” International Journal of Advanced Computer Science and Applications, vol. 15, no. 1, 2024, doi: 10.14569/IJACSA.2024.0150134.

[34] N. Sathiparan, S. H. Wijekoon, P. Jeyananthan, and D. N. Subramaniam, “Prediction of characteristics of pervious concrete by machine learning technique using mix parameters and non-destructive test measurements,” Nondestructive Testing and Evaluation, vol. 41, no. 1, pp. 314–363, Jan. 2026, doi: 10.1080/10589759.2025.2468273.

[35] D. Powers, “Evaluation: From precision, recall and F-measure to ROC,” Journal of Machine Learning Technologies, 2011.