A Hybrid Deep Learning Framework for Stock Market Prediction Using PPO-Based Sentiment Analysis and Transductive LSTM

Abstract

Stock market prediction is a challenging task due to the highly volatile, nonlinear, and dynamic nature of financial markets. This work proposes a hybrid intelligent framework that integrates technical indicators, reinforcement learning–based sentiment analysis, and deep learning models to enhance stock price forecasting accuracy. The system begins with dataset preprocessing and quality validation, followed by the extraction of technical indicators such as Simple Moving Average (SMA), Relative Strength Index (RSI), and Moving Average Convergence Divergence (MACD) from the NIFTY-50 dataset. In addition, market sentiment information is extracted and classified using an Off-Policy Proximal Policy Optimization (PPO) based sentiment model combined with deep learning features. The sentiment classification results demonstrate strong performance, achieving 92.4% accuracy, 93.4% F1-score, and a G-Mean of 0.908, outperforming baseline models such as BiLSTM (88.1% accuracy) and BERT+CNN (86.3% accuracy). The reinforcement learning training process shows stable convergence, where the average reward increases from –0.18 to approximately 0.90, indicating effective policy learning. For stock price prediction, a Transductive Long Short-Term Memory (TLSTM) model is employed to capture long-term temporal dependencies in financial time series. Experimental results show that the TLSTM model achieves RMSE = 0.045, MAPE = 2.14%, and MAE = 0.031, outperforming traditional models such as ARIMA (RMSE = 0.082), SVM (RMSE = 0.069), and BiLSTM (RMSE = 0.064). The proposed framework also demonstrates significant improvements compared to baseline methods, achieving 22.41% reduction in RMSE, 15.08% reduction in MAPE, and 20.51% reduction in MAE. Statistical significance analysis further confirms the robustness of the proposed approach with p-values below 0.05 when compared with several baseline models.