Utilizing Machine Learning based on LLM for Arabic Sentiment Analysis in Assessing User Satisfaction with Mobile Banking Apps: A Case Study of Yemeni Banks

Abstract

The development of large language models (LLMs) that are optimized to obey human commands is a significant advancement in the field of artificial intelligence (AI). One such model is ChatGPT (Chat Generative Pre-trained Transformer) from OpenAI, which has shown itself to be an extremely powerful tool for a variety of tasks such as conversation production, code debugging, and answering questions. Despite the fact that these models are praised for their multilingualism, little research has been done on how well they can analyze sentiment, especially in Arabic. We intend to close this gap by thoroughly assessing ChatGPT’s sentiment analysis skills, particularly with regard to Arabic text, in light of this constraint. When developing applications, recognizing the quality of the application and satisfying user needs are essential. Understanding user requirements is crucial to improving the quality of programs. The use of application review-based sentiment analysis (SA) is one efficient method for accomplishing this. The purpose of this study was to evaluate consumer perceptions of mobile banking apps so that they could be updated and maintained appropriately. Since mobile banking apps are now a necessary part of people’s lives, it is essential to examine user reviews of these apps for SA purposes. User reviews of banking mobile apps on the Google Play Store provided the dataset used in this study. We suggest a new active labeling technique for ChatGPT and examine the effects of using the ChatGPT variants for Arabic sentiment analysis (ASA). Using the accuracy, recall, precision, and F-score measures, we assess the performance of four machine learning (ML) approaches: Naive Bayes (NB), K-Nearest Neighbors (K-NN), Support Vector Machine (SVM), and Random Forest (FR). Additionally, we contrast six approaches to data labeling: manual human labeling, ChatGPT labeling by Assistant-Poe, ChatGPT labeling by Bing-Edge, ChatGPT labeling by Assistant-Poe with humans, ChatGPT labeling by Bing-Edge with humans, and ChatGPT labeling by Assistant-Poe with Bing-Edge. Using different Bing-Edge models for ASA, our experimental results demonstrate that the NB approach performed the best, with an accuracy of 91.22%, recall of 89.62%, precision of 88.90%, and F-score of 89.26%. Additionally, we contrast two approaches of data labeling for ASA: human labeling and labeling with Bard Google. Using Bard Google models for ASA, our experimental results demonstrate that the Naive Bayes technique outperformed the others, attaining an accuracy of 98.07%. Furthermore, when compared to alternative labeling techniques, our suggested active labeling strategy with ChatGPT produced greater accuracy. Our research indicates that our suggested active labeling method and the NB technique with multiple Bing-Edge models are useful strategies for ASA using ChatGPT. Our research provides important insights into efficient methods for this endeavor and advances the field of sentiment analysis in Arabic literature. Additionally, compared to other labeling techniques, our suggested active labeling method using Bard Google produced greater accuracy. According to the proposed study, our suggested active labeling strategy and the Naive Bayes technique with Bard Google models are both efficient methods for Arabic sentiment analysis utilizing Bard Google.