Big Data Analytics for Real-Time Market Trend Prediction

  • Authors

    • Dr. Katrina Keif Faculty of Information Technology, Brno University of Technology, Czech Republic. Author

    DOI:

    https://doi.org/10.67228/30713498/IJADSMC-2020PI2D8H

    Published 03-04-2020

  • Big Data Analytics, Real-Time Analytics, Market Trend Prediction, Stream Processing, Machine Learning, Predictive Modeling

    Issue

    Vol. 3 No. 1 (2020)

    Section

    Articles

    How to Cite

    [1]
    K. Keif, “Big Data Analytics for Real-Time Market Trend Prediction”, IJADSMC, vol. 3, no. 1, pp. 01–14, Mar. 2020, doi: 10.67228/30713498/IJADSMC-2020PI2D8H.

  • Abstract

    The rise of the digital platform the electronic trading system, the use of social media, and the Internet-connection devices have created an unprecedented amount of data, the speed, and diversity of market-related information. The time-honored approaches to market analysis, which the use of priori data and regular reports, are becoming increasingly insufficient in the sense that they do not reflect the dynamic and non-linear character of contemporary financial and commercial markets. In this regard, the essence of Big Data Analytics (BDA) has been found to be a pivotal mechanism in the art of predicting market trends in real time so that organizations are able to derive actionable information out of the monolithic, heterogenous and highly dynamic streams of data. The article is a SWOT analysis of predicting market trends in real-time that uses the landscape of analytics based on Big Data. The paper is a synthesis of data engineering pipelines, distributed stream processing architecture, machine learning models, and predictive analytics methods into one single methodological framework. It focuses on high-frequency, unstructured data, concept drift and low-latency decision-making. The suggested framework integrates the batch-stream hybrid processing, feature learning on structured and unstructured sources, and adaptive predictors that are able to adapt to the changing market forces. The literature survey will be conducted thoroughly and will analyze the previous studies concerning market prediction, time-series forecasting, sentiment analysis, the scalable platform of analytics with an emphasis on all major drawbacks that can be found in the areas of scalability, adaptability, and real-time responsiveness. Based on these observations, the paper suggests a framework of layered architecture of real-time market trend forecasting, which is promoted by proper formal mathematical modelling and algorithm design. As it has been proven through experimental assessment, the specified approach proves to be more accurate and responsive in terms of prediction, as well as more robust in contrast with traditional analytical models. The results of this study demonstrate the radical potential of Big Data Analytics in prohibitory time commercial understanding and making of decisions. The paper wraps up by covering the practical implications and limitations and future research, such as explainable AI, federated learning, and privacy-preserving analytics, as a next-generation market prediction system.

  • References

    [1] G. E. P. Box and G. M. Jenkins, Time Series Analysis: Forecasting and Control, 3rd ed. Englewood Cliffs, NJ, USA: Prentice-Hall, 1994.

    [2] R. S. Tsay, Analysis of Financial Time Series, 3rd ed. Hoboken, NJ, USA: Wiley, 2010.

    [3] C. Chatfield, the Analysis of Time Series: An Introduction, 6th ed. Boca Raton, FL, USA: CRC Press, 2004.

    [4] B. G. Malkiel, A Random Walk Down Wall Street, Rev. ed. New York, NY, USA: W. W. Norton & Company, 2012.

    [5] R. N. Elliott and J. Magee, Technical Analysis of Stock Trends, 9th ed. Boca Raton, FL, USA: CRC Press, 2005.

    [6] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed. New York, NY, USA: Springer, 2009.

    [7] V. Vapnik, The Nature of Statistical Learning Theory, 2nd ed. New York, NY, USA: Springer, 2000.

    [8] J. Patel, S. Shah, P. Thakkar, and K. Kotecha, “Predicting stock and stock price index movement using trend deterministic data preparation and machine learning techniques,” Expert Systems with Applications, vol. 42, no. 1, pp. 259–268, 2015.

    [9] Y. Fischer and J. Krauss, “Deep learning with long short-term memory networks for financial market predictions,” European Journal of Operational Research, vol. 270, no. 2, pp. 654–669, 2018.

    [10] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997.

    [11] Gandomi and M. Haider, “Beyond the hype: Big data concepts, methods, and analytics,” International Journal of Information Management, vol. 35, no. 2, pp. 137–144, 2015.

    [12] M. Zaharia et al., “Discretized streams: Fault-tolerant streaming computation at scale,” in Proc. 24th ACM Symp. Operating Systems Principles (SOSP), 2013, pp. 423–438.

    [13] T. Akidau et al., “The dataflow model: A practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing,” Proc. VLDB Endowment, vol. 8, no. 12, pp. 1792–1803, 2015.

    [14] N. Marz and J. Warren, Big Data: Principles and Best Practices of Scalable Realtime Data Systems. Shelter Island, NY, USA: Manning Publications, 2015.

  • Downloads