Building Scalable Data Infrastructure for Generative AI Models: Challenges and Solutions

  • Authors

    • Dr. Kwame Nkosi Department of Economics, University of Ghana, Accra, Ghana. Author

    DOI:

    https://doi.org/10.67228/30713315/IJAIDT-2018PII9F3R

    Published 08-04-2018

  • Generative AI, Data Infrastructure, Scalability, Data Engineering, Cloud Computing, Real-Time Data Processing, AI Workloads

    Issue

    Vol. 1 No. 2 (2018)

    Section

    Articles

    How to Cite

    [1]
    K. Nkosi, “Building Scalable Data Infrastructure for Generative AI Models: Challenges and Solutions”, IJAIDT, vol. 1, no. 2, pp. 01–08, Aug. 2018, doi: 10.67228/30713315/IJAIDT-2018PII9F3R.

  • Abstract

    The rapid advancement of Generative AI models has underscored the necessity for robust and scalable data infrastructures capable of managing vast datasets and complex computational requirements. This paper explores the unique challenges encountered in building such infrastructures, including data acquisition, storage, processing, and real-time access. We analyze existing solutions and propose best practices for designing architectures that ensure efficiency, scalability, and reliability. By examining case studies and current industry practices, the paper provides a comprehensive framework for developing data infrastructures tailored to the demands of Generative AI applications.

  • References

    [1] Zaharia, M., Chowdhury, M., Franklin, M. J., Shenker, S., & Stoica, I. (2010). Spark: Cluster computing with working sets. Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing, 10(10), 1–7.

    [2] Dean, J., & Ghemawat, S. (2008). MapReduce: Simplified data processing on large clusters. Communications of the ACM, 51(1), 107–113.

    [3] White, T. (2015). Hadoop: The definitive guide (4th ed.). O’Reilly Media.

    [4] Shvachko, K., Kuang, H., Radia, S., & Chansler, R. (2010). The Hadoop distributed file system. In 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (pp. 1–10).

    [5] Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang, C., & Zhang, Z. (2015). MXNet: A flexible and efficient machine learning library for heterogeneous distributed systems. arXiv preprint arXiv:1512.01274.

    [6] Abadi, M., Agarwal, A., Barham, P., et al. (2016). TensorFlow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467.

    [7] Meng, X., Bradley, J., Yavuz, B., et al. (2016). MLlib: Machine learning in Apache Spark. Journal of Machine Learning Research, 17(34), 1–7.

    [8] Vavilapalli, V. K., Murthy, A. C., Douglas, C., et al. (2013). Apache Hadoop YARN: Yet another resource negotiator. In Proceedings of the 4th Annual Symposium on Cloud Computing (pp. 1–16).

    [9] Li, M., Andersen, D. G., Park, J. W., et al. (2014). Scaling distributed machine learning with the parameter server. In 11th USENIX Symposium on Operating Systems Design and Implementation (pp. 583–598).

    [10] Crankshaw, D., Wang, X., Zhou, G., et al. (2017). Clipper: A low-latency online prediction serving system. In 14th USENIX Symposium on Networked Systems Design and Implementation (pp. 613–627).

  • Downloads