Smart Data Indexing Methods for Accelerated Query Processing

  • Authors

    • Ajay Krishnan Technical Lead, Tech Mahindra, India. Author

    DOI:

    https://doi.org/10.67228/30715717/IJDEIC-2019PI6P2M

    Published 05-08-2019

  • Smart Indexing, Query Processing, B-tree, Bitmap Index, Adaptive Indexing, Data Retrieval, Database Optimization, NoSQL, Big Data, Query Acceleration

    Issue

    Vol. 2 No. 1 (2019)

    Section

    Articles

    How to Cite

    [1]
    A. Krishnan, “Smart Data Indexing Methods for Accelerated Query Processing”, IJDEIC, vol. 2, no. 1, pp. 01–14, May 2019, doi: 10.67228/30715717/IJDEIC-2019PI6P2M.

  • Abstract

    The rapid growth of data from sources such as transactional systems, social media, IoT, and enterprise applications has created a strong need for efficient data retrieval systems. Traditional query processing methods often struggle with performance in large-scale, distributed, and heterogeneous environments. This paper examines smart data indexing techniques designed to improve query performance by reducing data access time and search complexity. It reviews traditional methods like B-trees, hash indexing, and bitmap indexing, and explores advanced approaches such as adaptive, multi-dimensional, and hybrid indexing. These techniques aim to optimize query response time, minimize disk I/O, and adapt to changing workloads. The study highlights the importance of indexing in DBMS, data warehouses, and big data platforms like Hadoop and NoSQL systems, while also addressing challenges such as scalability, storage overhead, and maintenance. Experimental results show that smart indexing methods significantly enhance query performance—for instance, adaptive indexing can reduce latency by up to 45%. Overall, the paper emphasizes the need for intelligent, self-optimizing indexing systems and suggests future directions including machine learning-based indexing for real-time analytics.

  • References

    [1] R. Bayer and E. McCreight, “Organization and Maintenance of Large Ordered Indexes,” Acta Informatica, vol. 1, no. 3, pp. 173–189, 1972.

    [2] D. Comer, “The Ubiquitous B-Tree,” ACM Computing Surveys, vol. 11, no. 2, pp. 121–137, 1979.

    [3] T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein, Introduction to Algorithms, 3rd ed. Cambridge, MA, USA: MIT Press, 2009.

    [4] W. Litwin, “Linear Hashing: A New Tool for File and Table Addressing,” in Proc. VLDB, 1980, pp. 212–223.

    [5] P.-Å. Larson, “Dynamic Hash Tables,” Communications of the ACM, vol. 31, no. 4, pp. 446–457, 1988.

    [6] P. O’Neil and D. Quass, “Improved Query Performance with Variant Indexes,” in Proc. ACM SIGMOD, 1997, pp. 38–49.

    [7] C.-Y. Chan and Y. E. Ioannidis, “Bitmap Index Design and Evaluation,” in Proc. ACM SIGMOD, 1998, pp. 355–366.

    [8] D. J. DeWitt and J. Gray, “Parallel Database Systems: The Future of High Performance Database Systems,” Communications of the ACM, vol. 35, no. 6, pp. 85–98, 1992.

    [9] S. Idreos, M. L. Kersten, and S. Manegold, “Database Cracking,” in Proc. CIDR, 2007, pp. 68–78.

    [10] S. Idreos et al., “MonetDB: Two Decades of Research in Column-oriented Database Architectures,” IEEE Data Engineering Bulletin, vol. 35, no. 1, pp. 40–45, 2012.

    [11] A. Guttman, “R-Trees: A Dynamic Index Structure for Spatial Searching,” in Proc. ACM SIGMOD, 1984, pp. 47–57.

    [12] J. L. Bentley, “Multidimensional Binary Search Trees Used for Associative Searching,” Communications of the ACM, vol. 18, no. 9, pp. 509–517, 1975.

    [13] G. Navarro, “A Guided Tour to Approximate String Matching,” ACM Computing Surveys, vol. 33, no. 1, pp. 31–88, 2001.

    [14] S. Brin and L. Page, “The Anatomy of a Large-Scale Hypertextual Web Search Engine,” Computer Networks, vol. 30, no. 1–7, pp. 107–117, 1998.

    [15] J. Dean and S. Ghemawat, “MapReduce: Simplified Data Processing on Large Clusters,” in Proc. OSDI, 2004, pp. 137–150.

  • Downloads