Position and Language as Diagnostic Variables: A Controlled Invariance Test of Sentiment Classifiers on Indonesian-English Code-Switched Reviews

Kennedy Lie; Handri Santoso

doi:10.24252/instek.v11i2.71617

Authors

Kennedy Lie Pradita University https://orcid.org/0009-0000-7520-9720
Handri Santoso Pradita University https://orcid.org/0000-0002-3265-8179

DOI:

https://doi.org/10.24252/instek.v11i2.71617

Keywords:

sentiment analysis, code-switched data, behavioral invariance testing, deep learning transformers, multilingual transformers

Abstract

Sentiment classifiers are usually judged by accuracy or F1-score, which cannot show whether a model reads the sentiment cue or relies on the surrounding text. Code-switched text, which mixes languages within a sentence, makes this harder to judge. This study proposes a controlled behavioural invariance test in which one sentiment cue is moved across three positions (beginning, middle, end) and written in two languages (Indonesian, English) in 272 reviews, giving 1,632 minimal-pair variants. Logistic Regression, Linear SVC, BERT, mBERT, XLM-RoBERTa and IndoBERT were compared using Prediction Consistency Rate, Prediction Flip Rate and McNemar tests with Holm correction, alongside standard metrics. The two TF-IDF models were largely insensitive to cue position and language but still flipped 5.1% and 1.5% of predictions, and deleting the frequent token “aplikasi” changed their output margins nearly as much as deleting the sentiment cue. BERT and mBERT made more errors on Indonesian than on English cues (adjusted p = 0.014 and 0.031). XLM-RoBERTa was unaffected by the token on the reported split and, over five repeated splits, had the highest mean accuracy and PCR and the lowest PFR. On this dataset, strong standard metrics did not guarantee consistent predictions when the cue was moved or translated.

Downloads

Download data is not yet available.

References

[1] B. Liu, Sentiment analysis : mining opinions, sentiments, and emotions. Cambridge University Press, 2020. doi: 10.1017/9781108639286.

[2] S. Poplack, "Sometimes I'll start a sentence in Spanish Y TERMINO EN ESPANOL: toward a typology of code-switching," Linguistics, vol. 18, no. 7-8, 1980, doi: 10.1515/ling.1980.18.7-8.581.

[3] A. F. Hidayatullah, R. A. Apong, D. T. C. Lai, and A. Qazi, "Pre-trained language model for code-mixed text in Indonesian, Javanese, and English using transformer," Soc. Netw. Anal. Min., vol. 15, no. 1, Dec. 2025, doi: 10.1007/s13278-025-01444-9.

[4] G. Winata, A. F. Aji, Z. X. Yong, and T. Solorio, "The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challenges," in Findings of the Association for Computational Linguistics: ACL 2023, Stroudsburg, PA, USA: Association for Computational Linguistics, 2023, pp. 2936-2978. doi: 10.18653/v1/2023.findings-acl.185.

[5] A. F. Hidayatullah, R. A. Apong, D. T. C. Lai, and A. Qazi, "Corpus creation and language identification for code-mixed Indonesian-Javanese-English Tweets," PeerJ Comput. Sci., vol. 9, p. e1312, 2023, doi: 10.7717/peerj-cs.1312.

[6] G. I. Winata, S. Cahyawijaya, Z. Liu, Z. Lin, A. Madotto, and P. Fung, "Are Multilingual Models Effective in Code-Switching?," in Proceedings of the Fifth Workshop on Computational Approaches to Linguistic Code-Switching, Stroudsburg, PA, USA: Association for Computational Linguistics, Jul. 2021, pp. 142-153. doi: 10.18653/v1/2021.calcs-1.20.

[7] S. Khanuja, S. Dandapat, A. Srinivasan, S. Sitaram, and M. Choudhury, "GLUECoS: An Evaluation Benchmark for Code-Switched NLP," in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Stroudsburg, PA, USA: Association for Computational Linguistics, 2020, pp. 3575-3585. doi: 10.18653/v1/2020.acl-main.329.

[8] P. Patwa et al., "SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets," in Proceedings of the Fourteenth Workshop on Semantic Evaluation, Stroudsburg, PA, USA: International Committee for Computational Linguistics, 2020, pp. 774-790. doi: 10.18653/v1/2020.semeval-1.100.

[9] S. Chanda, A. Mishra, and S. Pal, "Sentiment analysis of code-mixed Dravidian languages leveraging pretrained model and word-level language tag," Natural Language Processing, vol. 31, no. 2, pp. 477-499, Mar. 2025, doi: 10.1017/nlp.2024.30.

[10] E. Yulianti, A. Kurnia, M. Adriani, and Y. S. Duto, "Normalisation of Indonesian-English Code-Mixed Text and its Effect on Emotion Classification," International Journal of Advanced Computer Science and Applications, vol. 12, no. 11, 2021, doi: 10.14569/IJACSA.2021.0121177.

[11] L. W. Astuti, Y. Sari, and Suprapto, "Code-Mixed Sentiment Analysis using Transformer for Twitter Social Media Data," International Journal of Advanced Computer Science and Applications, vol. 14, no. 10, 2023, doi: 10.14569/IJACSA.2023.0141053.

[12] D. Sankoff and S. Poplack, "A formal grammar for code-switching," Paper in Linguistics, vol. 14, no. 1, pp. 3-45, Jan. 1981, doi: 10.1080/08351818109370523.

[13] C. Myers-Scotton, Duelling Languages: Grammatical Structure in Codeswitching. Oxford University Press, 1993. doi: 10.1093/oso/9780198240594.001.0001.

[14] M. T. Ribeiro, T. Wu, C. Guestrin, and S. Singh, "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList," in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Stroudsburg, PA, USA: Association for Computational Linguistics, 2020, pp. 4902-4912. doi: 10.18653/v1/2020.acl-main.442.

[15] R. T. McCoy, E. Pavlick, and T. Linzen, "Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference," in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Stroudsburg, PA, USA: Association for Computational Linguistics, 2019, pp. 3428-3448. doi: 10.18653/v1/P19-1334.

[16] K. Clark, U. Khandelwal, O. Levy, and C. D. Manning, "What Does BERT Look at? An Analysis of BERT's Attention," in Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Stroudsburg, PA, USA: Association for Computational Linguistics, 2019, pp. 276-286. doi: 10.18653/v1/W19-4828.

[17] A. Vaswani et al., "Attention Is All You Need," 2017, Accessed: May 15, 2026. [Online]. Available: http://arxiv.org/abs/1706.03762

[18] P. Rust, J. Pfeiffer, I. Vulic, S. Ruder, and I. Gurevych, "How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models," in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Stroudsburg, PA, USA: Association for Computational Linguistics, 2021, pp. 3118-3135. doi: 10.18653/v1/2021.acl-long.243.

[19] Y. Aliyu, A. Sarlan, K. U. Danyaro, A. S. B. A. Rahman, and M. Abdullahi, "Sentiment Analysis in Low-Resource Settings: A Comprehensive Review of Approaches, Languages, and Data Sources," IEEE Access, vol. 12, pp. 66883-66909, 2024, doi: 10.1109/ACCESS.2024.3398635.

[20] M. K. Nazir, M. Bilal, and S. C. Shongwe, "Sentiment analysis for code-mixed low-resource languages: a systematic review of approaches, techniques, applications, challenges, and future directions," Soc. Netw. Anal. Min., vol. 16, no. 1, p. 47, Mar. 2026, doi: 10.1007/s13278-026-01588-2.

[21] G. I. Winata et al., "NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languages," in Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, Stroudsburg, PA, USA: Association for Computational Linguistics, 2023, pp. 815-834. doi: 10.18653/v1/2023.eacl-main.57.

[22] G. Aguilar, S. Kar, and T. Solorio, "LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation," in Proceedings of the Twelfth Language Resources and Evaluation Conference, 2020, pp. 1803-1813.

[23] B. Wilie et al., "IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding," in Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, Stroudsburg, PA, USA: Association for Computational Linguistics, 2020, pp. 843-857. doi: 10.18653/v1/2020.aacl-main.85.

[24] F. Koto, J. H. Lau, and T. Baldwin, "IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initialization," in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Stroudsburg, PA, USA: Association for Computational Linguistics, 2021, pp. 10660-10668. doi: 10.18653/v1/2021.emnlp-main.833.

[25] E. Hlavnova and S. Ruder, "Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features," in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Stroudsburg, PA, USA: Association for Computational Linguistics, 2023, pp. 7181-7198. doi: 10.18653/v1/2023.acl-long.396.

[26] M. Gardner et al., "Evaluating Models' Local Decision Boundaries via Contrast Sets," in Findings of the Association for Computational Linguistics: EMNLP 2020, Stroudsburg, PA, USA: Association for Computational Linguistics, 2020, pp. 1307-1323. doi: 10.18653/v1/2020.findings-emnlp.117.

[27] D. Kaushik, E. Hovy, and Z. C. Lipton, "Learning the Difference that Makes a Difference with Counterfactually-Augmented Data," Feb. 2020, [Online]. Available: http://arxiv.org/abs/1909.12434

[28] S. Tan and S. Joty, "Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglots," in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Stroudsburg, PA, USA: Association for Computational Linguistics, 2021, pp. 3596-3616. doi: 10.18653/v1/2021.naacl-main.282.

[29] S. D. Das, A. Basak, S. Mandal, and D. Das, "AdvCodeMix: Adversarial Attack on Code-Mixed Data," in ACM International Conference Proceeding Series, Association for Computing Machinery, Jan. 2022, pp. 125-129. doi: 10.1145/3493700.3493705.

[30] M. F. Adilazuarda, S. Cahyawijaya, G. I. Winata, P. Fung, and A. Purwarianti, "IndoRobusta: Towards Robustness Against Diverse Code-Mixed Indonesian Local Languages," in Proceedings of the First Workshop on Scaling Up Multilingual Evaluation, Stroudsburg, PA, USA: Association for Computational Linguistics, 2022, pp. 25-34. doi: 10.18653/v1/2022.sumeval-1.5.

[31] S. B. T. Krishnaram, Y. Hu, and K. Saravanan, "When Surface Form Changes Moderation Decisions: A Paired Study of Code-Mixed Workflow Instability," Jun. 2026, [Online]. Available: http://arxiv.org/abs/2606.05654

[32] R. Geirhos et al., "Shortcut learning in deep neural networks," Nat. Mach. Intell., vol. 2, no. 11, pp. 665-673, Nov. 2020, doi: 10.1038/s42256-020-00257-z.

[33] S. Jain and B. C. Wallace, "Attention is not Explanation," May 2019, [Online]. Available: http://arxiv.org/abs/1902.10186

[34] S. Wiegreffe and Y. Pinter, "Attention is not not Explanation," Sep. 2019, [Online]. Available: http://arxiv.org/abs/1908.04626

[35] A. Valdivia, M. V. Luzon, E. Cambria, and F. Herrera, "Consensus vote models for detecting and filtering neutrality in sentiment analysis," Information Fusion, vol. 44, pp. 126-135, Nov. 2018, doi: 10.1016/j.inffus.2018.03.007.

[36] D. Rajput, W. J. Wang, and C. C. Chen, "Evaluation of a decided sample size in machine learning applications," BMC Bioinformatics, vol. 24, no. 1, Dec. 2023, doi: 10.1186/s12859-023-05156-9.

[37] G. Salton and C. Buckley, "Term-weighting approaches in automatic text retrieval," Inf. Process. Manag., vol. 24, no. 5, pp. 513-523, Jan. 1988, doi: 10.1016/0306-4573(88)90021-0.

[38] A. M. Bakare, K. S. M. Anbananthen, S. Muthaiyah, J. Krishnan, and S. Kannan, "Punctuation Restoration with Transformer Model on Social Media Data," Applied Sciences (Switzerland), vol. 13, no. 3, Feb. 2023, doi: 10.3390/app13031685.

[39] O. M. Alyasiri and Y. N. Cheah, "Multi-Class Text Classification using Machine Learning Techniques," Engineering, Technology and Applied Science Research, vol. 15, no. 3, pp. 22598-22604, Jun. 2025, doi: 10.48084/etasr.9994.

[40] T. Joachims, "Text categorization with Support Vector Machines: Learning with many relevant features," in Machine Learning: ECML-98, C. Nedellec and C. Rouveirol, Eds., Berlin, Heidelberg: Springer Berlin Heidelberg, 1998, pp. 137-142.

[41] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," in Proceedings of the 2019 Conference of the North, Stroudsburg, PA, USA: Association for Computational Linguistics, 2019, pp. 4171-4186. doi: 10.18653/v1/N19-1423.

[42] T. Pires, E. Schlinger, and D. Garrette, "How Multilingual is Multilingual BERT?," in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Stroudsburg, PA, USA: Association for Computational Linguistics, 2019, pp. 4996-5001. doi: 10.18653/v1/P19-1493.

[43] A. Conneau et al., "Unsupervised Cross-lingual Representation Learning at Scale," in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Stroudsburg, PA, USA: Association for Computational Linguistics, 2020, pp. 8440-8451. doi: 10.18653/v1/2020.acl-main.747.

[44] F. Koto, A. Rahimi, J. H. Lau, and T. Baldwin, "IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP," in Proceedings of the 28th International Conference on Computational Linguistics, Stroudsburg, PA, USA: International Committee on Computational Linguistics, 2020, pp. 757-770. doi: 10.18653/v1/2020.coling-main.66.

[45] V. B. Nguyen, C. Seifert, and J. Schlotterer, "CEval: A Benchmark for Evaluating Counterfactual Text Generation," in Proceedings of the 17th International Natural Language Generation Conference, Stroudsburg, PA, USA: Association for Computational Linguistics, 2024, pp. 55-69. doi: 10.18653/v1/2024.inlg-main.6.

[46] Q. McNemar, "Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages," Psychometrika, vol. 12, no. 2, pp. 153-157, Jun. 1947, doi: 10.1007/BF02295996.

[47] S. Holm, "A Simple Sequentially Rejective Multiple Test Procedure," Scandinavian Journal of Statistics, vol. 6, no. 2, pp. 65-70, 1979.

[48] C. D. Manning, P. Raghavan, and H. Schutze, Introduction to Information Retrieval. Cambridge: Cambridge University Press, 2008. doi: 10.1017/CBO9780511809071.

[49] M. Mosbach, M. Andriushchenko, and D. Klakow, "On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines," Mar. 2021, [Online]. Available: http://arxiv.org/abs/2006.04884