International Journal of Data Engineering and Intelligent Computing
type: channel
language: en
publication name: International Journal of Data Engineering and Intelligent Computing
issn: 3071-5717
items: Retrieval-Augmented Intelligent Data Warehousing for Decision Support Systems
items: Hybrid AI and Digital Twin Architecture for Enterprise Commerce and Financial Transformation
items: Graph-Based Approaches to Complex Data Transformation Workflows
items: Explainable AI Techniques for Intelligent Data Pipeline Optimization
items: Large Language Model-Assisted Metadata Engineering for Enterprise Data Platforms
items: Graph-Based Data Engineering Models for Large-Scale Knowledge Discovery
items: Federated Data Engineering Frameworks for Privacy-Aware Cross-Enterprise Analytics
items: Autonomous Data Fabric Architectures for Enterprise-Wide Intelligent Computing
items: Intelligent Workflow Scheduling for Distributed Data Processing Systems
items: Automated Data Transformation Using Intelligent Rule-Based Systems
items: Adaptive Query Routing for High-Speed Distributed Data Systems
items: AI-Assisted Data Pipeline Orchestration for Scalable Analytics
items: Semantic Data Integration Techniques for Heterogeneous Data Sources
items: AI-Powered Data Engineering Frameworks for Next-Generation Analytics
items: Intelligent Metadata Discovery in Complex Data Ecosystems
items: Synthetic Data Generation Strategies for Pipeline Robustness
items: Unified Metadata Management Framework for Hybrid Cloud Data Analytics
items: Serverless Data Engineering: Leveraging Cloud Technologies for Cost-Effective Data Workflows
items: Multi-Agent Systems for Autonomous Data Pipeline Optimization in AI Workflows
items: Ontology-Guided Data Mesh Design for Decentralized Governance
items: Multi-Path Data Transmission for Enhancing Reliability in Wireless Sensor Networks
items: Multi-Tenant Data Architecture for Real-Time AI Workloads with Role-Based Compliance Enforcement
items: Optimized Data Lake Architectures for Scalable ML Workflows
items: Machine Learning-Driven IT Cost Prediction Models in Scalable Data Platforms
items: Integrating Lakehouse Architecture with Data Sovereignty-Compliant Big Data Workloads
items: Integration of DataOps with Automated Exploratory Data Analysis (EDA) Workflows
items: Metadata-Centric Semantic Layer Management in Multi-Domain Pipelines
items: Industrial Big Data Analytics: Real-Time Decision-Making in Manufacturing
items: Impact of Data Versioning on Longitudinal Analytical Model Performance
items: Hybrid Data Models for Industrial Internet of Things (IIoT) Analytics

Retrieval-Augmented Intelligent Data Warehousing for Decision Support Systems
type: item
description: Organizations are increasingly challenged by massive volumes of structured, semi-structured, and unstructured data generated from enterprise systems, cloud platforms, IoT devices, social media, and industrial applications. Traditional data warehouses are effective for historical analytics but lack the ability to provide semantic understanding, dynamic knowledge retrieval, and real-time decision support. This study proposes a Retrieval-Augmented Intelligent Data Warehousing (RAIDW) framework that integrates Retrieval-Augmented Generation (RAG), vector databases, knowledge graphs, semantic search, machine learning, and large language models with conventional data warehousing. The framework combines ETL pipelines, vector indexing, retrieval ranking, and transformer-based reasoning to deliver context-aware, explainable, and accurate analytical insights. Machine learning improves retrieval relevance, while feedback mechanisms continuously enhance system performance. Experimental results demonstrate significant improvements in retrieval precision, analytical accuracy, semantic consistency, query efficiency, and decision confidence compared to traditional OLAP-based systems. The proposed architecture supports diverse domains, including healthcare, finance, supply chain, manufacturing, cybersecurity, and education, while ensuring governance, scalability, transparency, and security. By grounding AI-generated responses in trusted enterprise knowledge, the framework minimizes hallucinations and enables reliable, intelligent enterprise decision support.
subject: Articles
  topic: section
subject: Retrieval-Augmented Generation (RAG)
  topic: keywords
subject: Intelligent Data Warehouse
  topic: keywords
subject: Decision Support System (DSS)
  topic: keywords
subject: Business Intelligence
  topic: keywords
subject: Enterprise Analytics
  topic: keywords
subject: Artificial Intelligence
  topic: keywords
subject: Machine Learning
  topic: keywords
subject: Vector Database
  topic: keywords
subject: Knowledge Retrieval
  topic: keywords
subject: Large Language Models
  topic: keywords
subject: Semantic Search
  topic: keywords
subject: Data Integration
  topic: keywords
subject: Explainable AI
  topic: keywords
creator: Seshagiri N
rights: Copyright (c) 2025
date: 2025-02-05
publication date: 2025-02-05
starting page: 01
ending page: 16
doi: 10.67228/30715717/IJDEIC-2025PI8M4W

Hybrid AI and Digital Twin Architecture for Enterprise Commerce and Financial Transformation
type: item
description: Enterprise commerce and financial ecosystems are being transformed by Artificial Intelligence (AI), Cloud Computing, Big Data, Blockchain, the Internet of Things (IoT), and intelligent automation. However, conventional enterprise information systems primarily support operational automation and often lack real-time predictive intelligence, adaptive decision-making, and integrated visibility across business processes. These limitations lead to data silos, inaccurate forecasting, operational inefficiencies, financial risks, and reduced organizational agility. This paper proposes a Hybrid AI and Digital Twin Architecture for Enterprise Commerce and Financial Transformation that integrates intelligent data acquisition, digital twin modeling, hybrid AI analytics, financial intelligence, enterprise optimization, and continuous learning into a unified computational framework. The architecture combines machine learning, deep learning, reinforcement learning, knowledge graphs, explainable AI, and optimization techniques with dynamic digital twins that continuously synchronize with data from ERP, CRM, IoT devices, cloud platforms, and financial systems. The proposed framework enables predictive analytics, business process simulation, anomaly detection, financial forecasting, automated risk assessment, and explainable decision support. Organizations can evaluate multiple operational scenarios within a virtual environment before implementation, thereby reducing uncertainty, improving resilience, and enhancing strategic planning. Mathematical models are also introduced to evaluate enterprise operational efficiency, digital twin synchronization, predictive intelligence, and financial transformation performance. The proposed architecture is expected to improve forecasting accuracy, optimize supply chain operations, strengthen fraud detection, reduce operational costs, enhance customer experience, support regulatory compliance, and promote sustainable enterprise growth. By integrati…
subject: Articles
  topic: section
subject: Hybrid Artificial Intelligence
  topic: keywords
subject: Digital Twin
  topic: keywords
subject: Enterprise Commerce
  topic: keywords
subject: Financial Transformation
  topic: keywords
subject: Machine Learning
  topic: keywords
subject: Digital Enterprise
  topic: keywords
subject: Intelligent Automation
  topic: keywords
subject: Predictive Analytics
  topic: keywords
subject: Business Intelligence
  topic: keywords
subject: Explainable AI
  topic: keywords
creator: Alan Bundy
creator: Karen Spärck Jones
rights: Copyright (c) 2021
date: 2021-06-05
publication date: 2021-06-05
starting page: 01
ending page: 18
doi: 10.67228/30715717/IJDEIC-2021PI7V1G

Graph-Based Approaches to Complex Data Transformation Workflows
type: item
description: Modern data workflows are increasingly complex, involving heterogeneous data sources, dynamic transformation logic, and stringent performance requirements. Traditional linear and script-based approaches often struggle to provide the modularity, scalability, and traceability required for these workflows. In this paper, we explore graph-based approaches to modeling and executing complex data transformation workflows. By representing transformation logic as graphs where nodes denote operations and edges represent data or control dependencies these methods offer a flexible and powerful paradigm for structuring, optimizing, and managing workflows. We present a comprehensive framework for graph-based workflow modeling, discuss optimization strategies, and examine real-world applications and systems that leverage this methodology. We also analyze the trade-offs and limitations of such approaches and suggest future research directions in dynamic and intelligent graph-based systems.
subject: Articles
  topic: section
subject: Graph-based modeling
  topic: keywords
subject: Data transformation workflows
  topic: keywords
subject: Workflow optimization
  topic: keywords
subject: DAGs (Directed Acyclic Graphs)
  topic: keywords
subject: Data pipelines
  topic: keywords
subject: ETL/ELT systems
  topic: keywords
subject: Data engineering
  topic: keywords
subject: Workflow orchestration
  topic: keywords
subject: Graph traversal
  topic: keywords
creator: Tony Hoare
rights: Copyright (c) 2022
date: 2022-10-06
publication date: 2022-10-06
starting page: 01
ending page: 09
doi: 10.67228/30715717/IJDEIC-2022PII4H3A

Explainable AI Techniques for Intelligent Data Pipeline Optimization
type: item
description: Modern enterprises rely on intelligent data pipelines to collect, process, transform, and analyze data from diverse sources such as cloud platforms, IoT devices, enterprise systems, and social media. Traditional optimization techniques, including rule-based scheduling and heuristic resource allocation, improve efficiency but struggle to adapt to dynamic workloads, changing resource availability, and evolving business requirements. Artificial Intelligence (AI) addresses these limitations through predictive analytics, adaptive scheduling, anomaly detection, and autonomous resource optimization. However, the opaque nature of many AI models reduces transparency, trust, and regulatory compliance. This paper proposes an Explainable AI (XAI)-based Intelligent Data Pipeline Optimization Framework that integrates data preprocessing, predictive analytics, explainability, and adaptive optimization. The framework continuously monitors pipeline performance, generates optimization recommendations, and provides human-interpretable explanations for AI-driven decisions using feature attribution and model interpretation techniques. An automated feedback mechanism enables continuous learning and improvement. Experimental evaluation demonstrates enhanced optimization accuracy, reliability, scalability, interpretability, and administrator trust with minimal impact on performance. The proposed framework provides a transparent and trustworthy approach for next-generation intelligent data engineering systems.
subject: Articles
  topic: section
subject: Explainable Artificial Intelligence (Xai)
  topic: keywords
subject: Intelligent Data Pipelines
  topic: keywords
subject: Data Pipeline Optimization
  topic: keywords
subject: Machine Learning
  topic: keywords
subject: Workflow Scheduling
  topic: keywords
subject: Predictive Analytics
  topic: keywords
subject: Feature Attribution
  topic: keywords
subject: Resource Optimization
  topic: keywords
subject: Data Engineering
  topic: keywords
subject: Trustworthy Ai
  topic: keywords
subject: Pipeline Automation
  topic: keywords
subject: Decision Support Systems
  topic: keywords
creator: Per Brinch Hansen
creator: Ole Secher Olesen
rights: Copyright (c) 2025
date: 2025-09-04
publication date: 2025-09-04
starting page: 01
ending page: 14
doi: 10.67228/30715717/IJDEIC-2025PII7D2P

Large Language Model-Assisted Metadata Engineering for Enterprise Data Platforms
type: item
description: Enterprise metadata is essential for data discovery, governance, integration, and analytics. Traditional metadata engineering relies on manual, rule-based approaches that struggle with dynamic, heterogeneous enterprise data across cloud, IoT, ERP, CRM, and data lake environments. This study proposes a Large Language Model-Assisted Metadata Engineering Framework (LLM-MEF) that automates metadata extraction, semantic enrichment, schema recommendation, lineage discovery, and governance validation using transformer-based LLMs, Retrieval-Augmented Generation (RAG), vector databases, and knowledge graphs. The framework improves metadata quality, semantic consistency, discoverability, governance compliance, and operational efficiency while reducing manual effort. Explainable AI and continuous feedback learning further enhance transparency and adaptive improvement. The proposed LLM-MEF provides a scalable and intelligent solution for enterprise metadata management, supporting modern data governance, AI, business intelligence, regulatory compliance, and digital transformation.
subject: Articles
  topic: section
subject: Large Language Models (LLMs)
  topic: keywords
subject: Metadata Engineering
  topic: keywords
subject: Enterprise Data Platforms
  topic: keywords
subject: Data Governance
  topic: keywords
subject: Knowledge Graphs
  topic: keywords
subject: Semantic Metadata
  topic: keywords
subject: Retrieval-Augmented Generation (RAG)
  topic: keywords
subject: Enterprise Data Catalog
  topic: keywords
subject: Artificial Intelligence
  topic: keywords
subject: Data Lineage
  topic: keywords
creator: David Wheeler
creator: Michael Gordon
rights: Copyright (c) 2025
date: 2025-12-05
publication date: 2025-12-05
starting page: 01
ending page: 17
doi: 10.67228/30715717/IJDEIC-2025PII5X8V

Graph-Based Data Engineering Models for Large-Scale Knowledge Discovery
type: item
description: Graph-based data engineering has become a powerful approach for managing and analyzing highly interconnected data across enterprise systems, IoT, social media, healthcare, finance, and scientific domains. Unlike traditional relational databases, graph-based models represent data as interconnected nodes and edges, enabling efficient relationship analysis, semantic understanding, and knowledge discovery. This paper surveys recent advances in graph databases, knowledge graphs, graph neural networks (GNNs), and distributed graph analytics, and proposes an integrated framework for scalable graph construction, semantic enrichment, graph analytics, and AI-driven knowledge extraction. The framework emphasizes scalability, semantic consistency, explainable AI, and continuous graph evolution. Experimental evaluation demonstrates improved relationship discovery, query performance, and knowledge extraction compared with conventional relational approaches, making the proposed framework suitable for intelligent applications in healthcare, cybersecurity, finance, smart manufacturing, and enterprise knowledge management.
subject: Articles
  topic: section
subject: Graph Data Engineering
  topic: keywords
subject: Knowledge Graph
  topic: keywords
subject: Graph Database
  topic: keywords
subject: Graph Neural Networks
  topic: keywords
subject: Large-Scale Analytics
  topic: keywords
subject: Knowledge Discovery
  topic: keywords
subject: Semantic Networks
  topic: keywords
subject: Artificial Intelligence
  topic: keywords
subject: Graph Mining
  topic: keywords
subject: Data Integration
  topic: keywords
creator: Mahabala H. N
rights: Copyright (c) 2025
date: 2025-08-03
publication date: 2025-08-03
starting page: 01
ending page: 16
doi: 10.67228/30715717/IJDEIC-2025PII3W9L

Federated Data Engineering Frameworks for Privacy-Aware Cross-Enterprise Analytics
type: item
description: The growing adoption of data-driven decision-making has improved operational intelligence, predictive analytics, and strategic planning across enterprises. However, privacy regulations, data sovereignty requirements, and competitive concerns often restrict direct data sharing between organizations. Federated Data Engineering (FDE) addresses these challenges by enabling collaborative analytics without transferring sensitive raw data. This paper presents a privacy-aware Federated Data Engineering framework that integrates federated learning, distributed data engineering, and secure model aggregation for cross-enterprise analytics. The framework supports decentralized data preprocessing, feature engineering, and encrypted parameter sharing while complying with regulations such as GDPR and HIPAA. It incorporates privacy-enhancing technologies, including differential privacy, secure multi-party computation, homomorphic encryption, and blockchain-based auditing, to ensure confidentiality, integrity, and transparency. The proposed architecture also improves scalability, communication efficiency, fault tolerance, and interoperability across heterogeneous enterprise environments. Experimental evaluation demonstrates enhanced collaborative analytics with reduced privacy risks and communication overhead. The framework provides a practical foundation for secure, trustworthy, and privacy-preserving cross-enterprise data collaboration in healthcare, finance, manufacturing, and other data-intensive industries.
subject: Articles
  topic: section
subject: Federated Data Engineering
  topic: keywords
subject: Cross-Enterprise Analytics
  topic: keywords
subject: Privacy Preservation
  topic: keywords
subject: Federated Learning
  topic: keywords
subject: Differential Privacy
  topic: keywords
subject: Secure Multi-Party Computation
  topic: keywords
subject: Distributed Data Engineering
  topic: keywords
subject: Data Security
  topic: keywords
subject: Privacy-Aware Artificial Intelligence
  topic: keywords
creator: Karen Sparck Jones
creator: Donald Michie
rights: Copyright (c) 2025
date: 2025-07-05
publication date: 2025-07-05
starting page: 01
ending page: 18
doi: 10.67228/30715717/IJDEIC-2025PII3J2Q

Autonomous Data Fabric Architectures for Enterprise-Wide Intelligent Computing
type: item
description: Modern enterprises generate massive volumes of data from cloud platforms, IoT devices, enterprise applications, social media, and AI systems, creating challenges in data integration, governance, scalability, security, and real-time analytics. Traditional data management approaches often struggle to handle these complex and distributed environments. This paper proposes an Autonomous Data Fabric (ADF) architecture that combines AI/ML, metadata-driven automation, knowledge graphs, intelligent orchestration, and policy-based governance to enable seamless, self-managing enterprise data ecosystems. The framework supports automated data discovery, semantic integration, adaptive workflows, continuous monitoring, and intelligent resource optimization while ensuring data quality, security, and compliance. Experimental results demonstrate that the proposed ADF significantly improves data integration efficiency, governance, analytics performance, operational cost, and decision-making compared to conventional systems. Its scalable and self-adaptive design supports hybrid cloud, multi-cloud, edge, and on-premises environments, making it a robust solution for enterprise digital transformation and next-generation intelligent data management.
subject: Articles
  topic: section
subject: Autonomous Data Fabric
  topic: keywords
subject: Enterprise Intelligent Computing
  topic: keywords
subject: Artificial Intelligence
  topic: keywords
subject: Machine Learning
  topic: keywords
subject: Metadata Management
  topic: keywords
subject: Knowledge Graphs
  topic: keywords
subject: Enterprise Data Integration
  topic: keywords
subject: Data Governance
  topic: keywords
subject: Intelligent Analytics
  topic: keywords
subject: Distributed Data Architecture
  topic: keywords
subject: Hybrid Cloud
  topic: keywords
subject: Multi-Cloud Computing
  topic: keywords
subject: Enterprise Automation
  topic: keywords
subject: Semantic Data Integration
  topic: keywords
creator: Narendra Karmarkar
rights: Copyright (c) 2025
date: 2025-10-03
publication date: 2025-10-03
starting page: 01
ending page: 16
doi: 10.67228/30715717/IJDEIC-2025PII2J6Y

Intelligent Workflow Scheduling for Distributed Data Processing Systems
type: item
description: The rapid growth of data-intensive applications in scientific computing, enterprise analytics, and cloud services has increased the demand for efficient distributed data processing systems. Traditional scheduling methods like FCFS, Round Robin, and heuristic approaches often fail to meet the dynamic and heterogeneous requirements of modern environments. This paper proposes an intelligent workflow scheduling framework that improves performance through adaptive decision-making, predictive analytics, and machine learning. The system dynamically allocates tasks based on resource availability, workflow dependencies, and historical execution data, enabling it to anticipate bottlenecks and reassign tasks proactively. It also incorporates resource heterogeneity modeling and dependency-aware scheduling to reduce idle time and optimize execution. Performance is evaluated using metrics such as makespan, throughput, resource utilization, and fault tolerance, showing significant improvements over traditional methods. The framework also addresses key challenges like load balancing, scalability, energy efficiency, and fault tolerance. Overall, the proposed approach enhances system efficiency and scalability while supporting integration with emerging technologies such as edge computing and hybrid cloud environments, paving the way for more autonomous and resilient distributed scheduling systems.
subject: Articles
  topic: section
subject: Distributed Systems
  topic: keywords
subject: Workflow Scheduling
  topic: keywords
subject: Intelligent Scheduling
  topic: keywords
subject: Resource Allocation
  topic: keywords
subject: Big Data Processing
  topic: keywords
subject: Load Balancing
  topic: keywords
subject: Optimization Algorithms
  topic: keywords
subject: Cloud Computing
  topic: keywords
creator: David Parnas
rights: Copyright (c) 2024
date: 2024-12-04
publication date: 2024-12-04
starting page: 01
ending page: 13
doi: 10.67228/30715717/IJDEIC-2024PII8H5A

Automated Data Transformation Using Intelligent Rule-Based Systems
type: item
description: This paper explores automated data transformation using intelligent rule-based systems to address the limitations of traditional ETL processes, which are often manual, rigid, and error-prone. The proposed approach integrates rule-based reasoning with metadata-driven transformation to handle complex data from heterogeneous sources such as structured, semi-structured, and unstructured data. The system features a modular architecture including data ingestion, rule definition, execution, and validation. It applies condition–action rules for tasks like normalization, filtering, aggregation, and enrichment, along with a feedback mechanism for continuous improvement. Experimental results show that the approach significantly reduces transformation time while maintaining accuracy and improving data quality. Challenges such as rule conflicts, scalability, and legacy integration are also addressed through strategies like rule prioritization and hybrid architectures. The study concludes that intelligent rule-based systems offer a scalable and efficient solution for modern data transformation in big data and real-time environments.
subject: Articles
  topic: section
subject: Automated Data Transformation
  topic: keywords
subject: Rule-Based Systems
  topic: keywords
subject: Etl Automation
  topic: keywords
subject: Data Integration
  topic: keywords
subject: Intelligent Systems
  topic: keywords
subject: Data Processing
  topic: keywords
subject: Semantic Mapping
  topic: keywords
creator: Tom DeMarco
rights: Copyright (c) 2024
date: 2024-11-09
publication date: 2024-11-09
starting page: 01
ending page: 12
doi: 10.67228/30715717/IJDEIC-2024PII7D1M

Adaptive Query Routing for High-Speed Distributed Data Systems
type: item
description: Adaptive query routing is essential in high-speed distributed data systems because static routing cannot effectively handle dynamic changes in network conditions, workloads, and system heterogeneity; instead, it dynamically selects optimal nodes or paths based on real-time factors such as latency, node availability, workload distribution, and past query performance, thereby improving efficiency, scalability, and reliability. Unlike early deterministic or heuristic methods, adaptive approaches use feedback mechanisms and probabilistic models to continuously update routing decisions, enhancing throughput and reducing query response time. The main strategies include centralized routing (with a global view but limited scalability and single-point failure risk), decentralized routing (better scalability but coordination challenges), and hybrid models (balancing both approaches). Additionally, techniques like caching, replication, and indexing help reduce unnecessary data transfer and improve execution speed, while mathematical cost models guide optimal routing decisions. Experimental results show that adaptive routing can improve query response time by 40–60% under dynamic workloads, making it a crucial component for modern distributed systems, especially in cloud computing and big data environments.
subject: Articles
  topic: section
subject: Adaptive Query Routing
  topic: keywords
subject: Distributed Systems
  topic: keywords
subject: Load Balancing
  topic: keywords
subject: Data Locality
  topic: keywords
subject: Query Optimization
  topic: keywords
subject: Network Latency
  topic: keywords
subject: Distributed Databases
  topic: keywords
subject: High-Speed Systems
  topic: keywords
creator: Norbert Wiener
creator: Fernando J. Corbató
creator: Douglas Engelbart
rights: Copyright (c) 2024
date: 2024-10-10
publication date: 2024-10-10
starting page: 01
ending page: 12
doi: 10.67228/30715717/IJDEIC-2024PII6M2K

AI-Assisted Data Pipeline Orchestration for Scalable Analytics
type: item
description: Modern enterprises face increasing demands for scalable and efficient data processing due to rapid data growth. Traditional data pipeline orchestration methods, which rely on static configurations and manual intervention, often lead to inefficiencies in resource use, latency, and fault tolerance. This paper proposes an AI-assisted orchestration framework that integrates machine learning techniques to enable dynamic scheduling, workload prediction, anomaly detection, and resource optimization. By leveraging reinforcement learning, supervised learning, and heuristic methods, the system adapts pipeline configurations in real time based on changing workloads and system conditions. The proposed architecture includes data ingestion modules, AI-driven orchestration engines, adaptive schedulers, and monitoring systems. A key contribution is an intelligent scheduling mechanism that improves execution efficiency and resource utilization. Experimental results show significant improvements over traditional systems, with up to 35% increase in processing efficiency and 25% reduction in latency. The study concludes that AI-driven orchestration is a promising approach for building scalable and autonomous data processing systems, with future work focusing on deeper integration of advanced learning models and real-time adaptability.
subject: Articles
  topic: section
subject: AI Orchestration
  topic: keywords
subject: Data Pipelines
  topic: keywords
subject: Scalable Analytics
  topic: keywords
subject: Machine Learning
  topic: keywords
subject: Workflow Scheduling
  topic: keywords
subject: Distributed Systems
  topic: keywords
subject: Predictive Optimization
  topic: keywords
creator: Joseph Weizenbaum
creator: Seymour Papert
rights: Copyright (c) 2024
date: 2024-08-08
publication date: 2024-08-08
starting page: 01
ending page: 13
doi: 10.67228/30715717/IJDEIC-2024PII3Q9V

Semantic Data Integration Techniques for Heterogeneous Data Sources
type: item
description: Semantic data integration has become essential in data-driven environments where organizations handle large volumes of heterogeneous data from diverse sources such as databases, web services, XML, sensors, and social media. Traditional integration techniques like schema matching and data warehousing are insufficient to address structural, syntactic, and semantic differences across distributed systems. To overcome these challenges, semantic approaches based on ontologies, knowledge representation, and reasoning have gained importance. This work focuses on ontology-based integration, schema alignment, and semantic mediation to enable interoperability. It also examines integration models such as Global-as-View (GAV), Local-as-View (LAV), and hybrid approaches, along with semantic web technologies like RDF, OWL, and SPARQL. A systematic integration methodology is proposed, involving preprocessing, semantic annotation, ontology matching, conflict resolution, and query transformation. Mathematical models are used to define similarity measures and mapping confidence. Experimental evaluation on benchmark datasets shows improved accuracy, precision, recall, data consistency, and query efficiency compared to traditional methods. Despite these advancements, challenges such as scalability, ontology development, and semantic ambiguity remain open research issues.
subject: Articles
  topic: section
subject: Semantic Data Integration
  topic: keywords
subject: Ontology Alignment
  topic: keywords
subject: Heterogeneous Data Sources
  topic: keywords
subject: RDF
  topic: keywords
subject: Owl
  topic: keywords
subject: Schema Mapping
  topic: keywords
subject: Data Interoperability
  topic: keywords
subject: Semantic Web
  topic: keywords
subject: Knowledge Representation
  topic: keywords
subject: Data Fusion
  topic: keywords
creator: Paul Baran
rights: Copyright (c) 2024
date: 2024-08-08
publication date: 2024-08-08
starting page: 01
ending page: 15
doi: 10.67228/30715717/IJDEIC-2024PII2Y4R

AI-Powered Data Engineering Frameworks for Next-Generation Analytics
type: item
description: The rapid growth of heterogeneous data sources—such as IoT devices, social media, enterprise systems, and cloud applications—has led to massive increases in data volume, velocity, and variety. Traditional rule-based and fixed data engineering pipelines are no longer sufficient to handle these complexities. This paper explores AI-driven data engineering frameworks designed to build scalable, adaptive, and intelligent data pipelines. By integrating AI techniques like machine learning, deep learning, and reinforcement learning, these frameworks enable automated data ingestion, intelligent transformation, anomaly detection, and predictive pipeline optimization. Unlike traditional batch-processing systems, modern architectures support hybrid and real-time streaming, improving efficiency and flexibility. The proposed approach introduces a layered architecture where each stage—ingestion, processing, storage, orchestration, and analytics—is enhanced with AI capabilities. Results show significant improvements, including up to 45% reduction in data errors and 60% increase in pipeline efficiency. Overall, AI-based data engineering represents a major advancement, paving the way for more intelligent, self-optimizing systems, with future directions including explainable AI, federated learning, and edge computing.
subject: Articles
  topic: section
subject: AI-Powered Data Engineering
  topic: keywords
subject: Machine Learning
  topic: keywords
subject: Data Pipelines
  topic: keywords
subject: ETL
  topic: keywords
subject: Big Data Analytics
  topic: keywords
subject: Distributed Systems
  topic: keywords
subject: Data Quality
  topic: keywords
subject: Predictive Analytics
  topic: keywords
subject: Intelligent Automation
  topic: keywords
subject: Next-Generation Analytics
  topic: keywords
creator: Niklaus Wirth
rights: Copyright (c) 2024
date: 2024-05-06
publication date: 2024-05-06
starting page: 01
ending page: 12
doi: 10.67228/30715717/IJDEIC-2024PI7F1Y

Intelligent Metadata Discovery in Complex Data Ecosystems
type: item
description: The rapid growth of diverse data sources has created complex data ecosystems characterized by high volume, velocity, and variety. Organizations increasingly rely on distributed systems, data lakes, and cloud storage, which introduce challenges in managing and utilizing metadata. Metadata plays a crucial role in data governance, integration, and understanding, but traditional methods are often manual and inadequate for dynamic environments. This paper explores early intelligent metadata discovery approaches, including rule-based, statistical, and machine learning techniques. These methods automate the identification and enhancement of metadata across structured, semi-structured, and unstructured data sources, improving data quality and accessibility. Key challenges such as schema differences, semantic ambiguity, scalability, and data silos are addressed. A hybrid framework combining rule-based extraction, probabilistic models, and metadata enrichment is proposed. Results show that intelligent discovery methods significantly improve metadata accuracy, completeness, and efficiency while reducing manual effort. The study highlights their foundational role in modern data governance and emphasizes the need for scalable, adaptive metadata systems.
subject: Articles
  topic: section
subject: Intelligent Metadata Discovery
  topic: keywords
subject: Data Ecosystems
  topic: keywords
subject: Metadata Management
  topic: keywords
subject: Data Integration
  topic: keywords
subject: Semantic Analysis
  topic: keywords
subject: Data Governance
  topic: keywords
subject: Machine Learning
  topic: keywords
subject: Data Lakes
  topic: keywords
subject: Technologies
  topic: keywords
creator: Herbert A. Simon
creator: Lotfi A. Zadeh
rights: Copyright (c) 2024
date: 2024-04-10
publication date: 2024-04-10
starting page: 01
ending page: 13
doi: 10.67228/30715717/IJDEIC-2024PI5L8N

Synthetic Data Generation Strategies for Pipeline Robustness
type: item
description: Machine learning pipelines are increasingly deployed in high-stakes domains where robustness against data inconsistencies, bias, and distributional shifts is crucial. However, real-world datasets often suffer from limitations such as data scarcity, class imbalance, and privacy constraints. Synthetic data generation has emerged as a promising strategy to overcome these challenges and enhance pipeline robustness. This paper presents a comprehensive survey and analysis of synthetic data generation techniques, classifying them by data modality, generation method, and application purpose. We explore how synthetic data contributes to the resilience of ML pipelines against failure modes such as concept drift, noise, and adversarial attacks. Through empirical case studies across domains, we demonstrate the practical benefits and limitations of integrating synthetic data into training and evaluation pipelines. Finally, we discuss ethical considerations and outline future research directions toward building more robust, fair, and privacy-preserving ML systems using synthetic data.
subject: Articles
  topic: section
subject: Synthetic Data
  topic: keywords
subject: Pipeline Robustness
  topic: keywords
subject: Data Augmentation
  topic: keywords
subject: Generative Models
  topic: keywords
subject: Adversarial Robustness
  topic: keywords
subject: Concept Drift
  topic: keywords
subject: Machine Learning Evaluation
  topic: keywords
subject: Data Imbalance
  topic: keywords
subject: Privacy-Preserving ML
  topic: keywords
subject: ML Fairness
  topic: keywords
creator: Dennis Ritchie
creator: Allen Newell
rights: Copyright (c) 2024
date: 2024-03-09
publication date: 2024-03-09
starting page: 01
ending page: 14
doi: 10.67228/30715717/IJDEIC-2024PI4R3T

Unified Metadata Management Framework for Hybrid Cloud Data Analytics
type: item
description: In the era of big data, hybrid cloud environments have become integral for data analytics, offering flexibility, scalability, and cost-efficiency. However, managing metadata across these complex, distributed systems remains a significant challenge. Metadata, which describes the data, its context, and its usage, plays a crucial role in improving data discovery, quality, and governance. In this paper, we propose a Unified Metadata Management Framework designed specifically for hybrid cloud data analytics. This framework integrates metadata from multiple cloud environments (public, private, and multi-cloud) into a cohesive, centralized system that facilitates efficient data management, governance, and analytics. We discuss the key components of the framework, including metadata repositories, metadata exchange layers, cataloging, and data lineage tracking. Additionally, the framework ensures security and privacy compliance while offering scalability and flexibility. We provide use cases from healthcare, finance, and e-commerce sectors to demonstrate its practical applications and evaluate its performance against traditional metadata management approaches. The proposed solution offers significant improvements in managing metadata across hybrid cloud infrastructures and lays the foundation for future innovations in cloud data analytics.
subject: Articles
  topic: section
subject: Hybrid Cloud
  topic: keywords
subject: Metadata Management
  topic: keywords
subject: Data Analytics
  topic: keywords
subject: Unified Framework
  topic: keywords
subject: Cloud Data Governance
  topic: keywords
subject: Metadata Repository
  topic: keywords
subject: Data Lineage
  topic: keywords
subject: Cloud Computing
  topic: keywords
subject: Data Privacy
  topic: keywords
subject: Cloud Integration
  topic: keywords
subject: Data Security
  topic: keywords
creator: Ken Iverson
rights: Copyright (c) 2024
date: 2024-02-12
publication date: 2024-02-12
starting page: 01
ending page: 10
doi: 10.67228/30715717/IJDEIC-2024PI1B7P

Serverless Data Engineering: Leveraging Cloud Technologies for Cost-Effective Data Workflows
type: item
description: Serverless data engineering has emerged as a transformative approach to handling data workflows, offering scalable, cost-efficient, and automated data processing in cloud environments. This paradigm eliminates infrastructure management, enabling data engineers to focus on analytics, data transformations, and insights rather than server maintenance. The shift from traditional data processing to serverless architectures is driven by cost savings, operational efficiency, and scalability. This paper explores serverless data engineering principles, architectures, benefits, and challenges, with a particular focus on real-world applications across industries. We analyze existing cloud-based serverless solutions, including AWS Lambda, Google Cloud Functions, and Azure Functions, for data ingestion, transformation, and analytics. A comparative study of serverless vs. traditional data pipelines is provided, highlighting cost reductions, performance improvements, and automation benefits. Furthermore, we propose a framework for optimizing serverless data workflows, integrating event-driven processing, automated scaling, and security considerations. The paper concludes with a discussion on the future of serverless data engineering and recommendations for effective implementation in organizations seeking cost-effective and efficient data strategies.
subject: Articles
  topic: section
subject: Serverless Computing
  topic: keywords
subject: Cloud Data Engineering
  topic: keywords
subject: Cost-Effective Data Workflows
  topic: keywords
subject: AWS Lambda
  topic: keywords
subject: Google Cloud Functions
  topic: keywords
subject: Event-Driven Processing
  topic: keywords
subject: Serverless Data Pipelines
  topic: keywords
subject: Automated Scaling
  topic: keywords
subject: Cloud Computing
  topic: keywords
subject: Data Transformation
  topic: keywords
creator: Peter Naur
creator: John Backus
creator: Donald Davies
rights: Copyright (c) 2024
date: 2024-06-10
publication date: 2024-06-10
starting page: 01
ending page: 09
doi: 10.67228/30715717/IJDEIC-2024PI9X6C

Multi-Agent Systems for Autonomous Data Pipeline Optimization in AI Workflows
type: item
description: The optimization of data pipelines is critical for enhancing the performance and efficiency of AI workflows, which often involve complex, heterogeneous, and dynamic data processing stages. Traditional approaches to pipeline optimization struggle to adapt autonomously to evolving workloads and system conditions. This paper proposes a novel multi-agent system (MAS) framework that enables autonomous optimization of data pipelines in AI workflows. Each agent is responsible for specific tasks such as data ingestion, transformation, scheduling, and resource management, and they collaborate through adaptive protocols to achieve global optimization objectives. We demonstrate the effectiveness of the proposed framework through extensive experiments, showing significant improvements in pipeline throughput, latency, and resource utilization compared to conventional methods. Our approach highlights the potential of MAS to bring intelligence, flexibility, and scalability to data pipeline management in AI systems.
subject: Articles
  topic: section
subject: Multi-Agent Systems
  topic: keywords
subject: Data Pipeline Optimization
  topic: keywords
subject: Autonomous Systems
  topic: keywords
subject: AI Workflows
  topic: keywords
subject: Distributed Data Processing
  topic: keywords
subject: Workflow Management
  topic: keywords
subject: Adaptive Scheduling
  topic: keywords
subject: Resource Optimization
  topic: keywords
creator: José María Troya
creator: Ramon López de Mántaras
rights: Copyright (c) 2023
date: 2023-08-02
publication date: 2023-08-02
starting page: 01
ending page: 10
doi: 10.67228/30715717/IJDEIC-2023PII1N7X

Ontology-Guided Data Mesh Design for Decentralized Governance
type: item
description: The rise of Data Mesh as a paradigm for large-scale data management has challenged the traditional monolithic data lake approach by promoting decentralized ownership, domain-oriented architecture, and self-serve infrastructure. However, without a unified semantic layer, decentralization can lead to fragmentation, inconsistency, and governance bottlenecks. This paper proposes an ontology-guided approach to Data Mesh design, leveraging domain ontologies to ensure semantic interoperability, governance automation, and federated data product discovery. By aligning domain-specific knowledge structures with data product metadata, we demonstrate how ontologies facilitate coherent data governance across distributed teams while preserving the autonomy of individual domains. We explore architectural patterns, governance workflows, and implementation considerations, and present a case study to illustrate the application of this approach in a real-world enterprise setting.
subject: Articles
  topic: section
subject: Data Mesh
  topic: keywords
subject: Ontology
  topic: keywords
subject: Decentralized Data Governance
  topic: keywords
subject: Semantic Interoperability
  topic: keywords
subject: Federated Data Architecture
  topic: keywords
subject: Knowledge Graph
  topic: keywords
subject: Metadata Management
  topic: keywords
subject: Data Products
  topic: keywords
subject: Data-as-a-Product
  topic: keywords
subject: Domain-Driven Design
  topic: keywords
creator: Seppo Linnainmaa
creator: Arto Salomaa
rights: Copyright (c) 2023
date: 2023-12-06
publication date: 2023-12-06
starting page: 01
ending page: 13
doi: 10.67228/30715717/IJDEIC-2023PII8P5R

Multi-Path Data Transmission for Enhancing Reliability in Wireless Sensor Networks
type: item
description: Wireless Sensor Networks (WSNs) have gained significant attention due to their wide range of applications, including environmental monitoring, healthcare, industrial automation, and military operations. The primary challenge in WSNs is to ensure reliable data transmission while maintaining energy efficiency and network longevity. Multi-path data transmission has emerged as a promising technique to enhance the reliability of WSNs by mitigating data loss, reducing congestion, and improving fault tolerance. This paper presents a comprehensive study on multi-path data transmission mechanisms in WSNs, analyzing their impact on network performance, energy consumption, and data reliability. We explore various multi-path routing protocols, including Disjoint Path Routing, Braided Path Routing, and Hybrid Approaches, and assess their effectiveness in different network scenarios. Additionally, we discuss the challenges associated with multi-path data transmission, such as path redundancy, interference, and increased computational overhead. Through extensive simulations and comparative analysis, we demonstrate that multi-path data transmission significantly enhances network reliability while ensuring optimal resource utilization. The findings of this study provide valuable insights for designing robust and efficient WSNs, thereby contributing to advancements in the field of wireless communication.
subject: Articles
  topic: section
subject: Multi-Path Data Transmission
  topic: keywords
subject: Wireless Sensor Networks (WSNs)
  topic: keywords
subject: Reliability
  topic: keywords
subject: Fault Tolerance
  topic: keywords
subject: Energy Efficiency
  topic: keywords
subject: Routing Protocols
  topic: keywords
subject: Network Longevity
  topic: keywords
subject: Data Redundancy
  topic: keywords
creator: Anatoly Kitov
rights: Copyright (c) 2023
date: 2023-11-04
publication date: 2023-11-04
starting page: 01
ending page: 13
doi: 10.67228/30715717/IJDEIC-2023PII6T2H

Multi-Tenant Data Architecture for Real-Time AI Workloads with Role-Based Compliance Enforcement
type: item
description: In the age of real-time AI workloads, scalable and secure data infrastructure is paramount—especially in multi-tenant environments where multiple users or organizations share resources. This paper presents a novel architectural blueprint for designing a multi-tenant data system optimized for real-time AI processing. It addresses the dual challenge of performance and compliance by introducing a layered, role-based compliance enforcement mechanism. This approach enables dynamic access control, data isolation, lineage tracking, and auditability without compromising on latency or throughput. Through practical design patterns and reference implementations, the paper outlines how to balance architectural efficiency with rigorous regulatory obligations such as GDPR, HIPAA, or SOC 2. We also examine case studies and real-world implementations to validate the proposed framework.
subject: Articles
  topic: section
subject: Multi-Tenant Architecture
  topic: keywords
subject: Real-Time AI Workloads
  topic: keywords
subject: Role-Based Access Control (RBAC)
  topic: keywords
subject: Data Compliance
  topic: keywords
subject: Data Governance
  topic: keywords
subject: Secure Data Sharing
  topic: keywords
subject: Streaming Data Architecture
  topic: keywords
subject: AI/ML Infrastructure
  topic: keywords
subject: Regulatory Enforcement
  topic: keywords
subject: Scalable Data Systems
  topic: keywords
creator: Per Brinch Hansen
creator: Børge Diderichsen
rights: Copyright (c) 2023
date: 2023-10-10
publication date: 2023-10-10
starting page: 01
ending page: 12
doi: 10.67228/30715717/IJDEIC-2023PII3A9M

Optimized Data Lake Architectures for Scalable ML Workflows
type: item
description: The exponential growth of data and the increasing demand for real-time, large-scale machine learning (ML) applications have challenged traditional data storage and processing architectures. Data lakes have emerged as a flexible and cost-effective solution for storing massive amounts of heterogeneous data. However, without optimization, data lakes can become inefficient, leading to performance bottlenecks in ML workflows. This paper presents an in-depth exploration of optimized data lake architectures tailored for scalable ML pipelines. We examine architectural best practices, including the separation of storage and compute, data versioning, metadata management, and the use of open table formats like Delta Lake, Apache Iceberg, and Apache Hudi. Furthermore, we propose a reference architecture that integrates modern orchestration tools, feature stores, and distributed compute engines to streamline the ML lifecycle from data ingestion to model deployment. Performance benchmarks and use cases demonstrate the proposed architecture's effectiveness in improving scalability, maintainability, and end-to-end ML throughput.
subject: Articles
  topic: section
subject: Data Lake Architecture
  topic: keywords
subject: Scalable Machine Learning
  topic: keywords
subject: Data Engineering
  topic: keywords
subject: Delta Lake
  topic: keywords
subject: Apache Iceberg
  topic: keywords
subject: Feature Store
  topic: keywords
subject: Metadata Management
  topic: keywords
subject: ML Pipeline
  topic: keywords
subject: Optimization
  topic: keywords
subject: Distributed Compute
  topic: keywords
subject: Lakehouse
  topic: keywords
creator: Sergey Lebedev
rights: Copyright (c) 2023
date: 2023-09-03
publication date: 2023-09-03
starting page: 01
ending page: 13
doi: 10.67228/30715717/IJDEIC-2023PII2W4D

Machine Learning-Driven IT Cost Prediction Models in Scalable Data Platforms
type: item
description: In modern IT ecosystems, accurately predicting operational costs is critical for budgeting, resource allocation, and service optimization. Traditional cost estimation models often fall short in environments characterized by high variability and scale. This paper presents a machine learning-driven approach to IT cost prediction, leveraging scalable data platforms to process large volumes of operational data in real time. We explore various regression and time series forecasting models trained on usage metrics and workload patterns, and demonstrate how distributed architectures (e.g., cloud-native platforms, big data processing engines) enhance both the accuracy and scalability of the prediction process. Experimental results show that the proposed models significantly outperform traditional baselines in terms of accuracy and adaptability, offering a viable pathway for proactive cost management in complex IT environments.
subject: Articles
  topic: section
subject: IT Cost Prediction
  topic: keywords
subject: Machine Learning
  topic: keywords
subject: Scalable Data Platforms
  topic: keywords
subject: Cloud Computing
  topic: keywords
subject: Time Series Forecasting
  topic: keywords
subject: Big Data Analytics
  topic: keywords
subject: Cost Optimization
  topic: keywords
subject: Resource Management
  topic: keywords
subject: Predictive Modeling
  topic: keywords
subject: Distributed Systems
  topic: keywords
creator: Zdzisław Pawlak
rights: Copyright (c) 2023
date: 2023-06-08
publication date: 2023-06-08
starting page: 01
ending page: 12
doi: 10.67228/30715717/IJDEIC-2023PI9D6L

Integrating Lakehouse Architecture with Data Sovereignty-Compliant Big Data Workloads
type: item
description: The increasing complexity of data governance and regulatory compliance in an era of distributed, global data systems has made data sovereignty a critical consideration for modern data architectures. This paper explores the integration of Lakehouse Architecture with data sovereignty-compliant frameworks to efficiently manage big data workloads across multiple jurisdictions. By combining the flexibility and scalability of data lakes with the management and query optimization capabilities of data warehouses, Lakehouse architecture offers a unified solution that ensures both agility and compliance. We delve into the challenges and strategies of enforcing data sovereignty in multi-region and multi-cloud environments while maintaining high performance and security in big data operations. Additionally, we examine best practices, tools, and frameworks that support data sovereignty in Lakehouse implementations, offering a comprehensive guide to building compliant data pipelines and analytics systems.
subject: Articles
  topic: section
subject: Lakehouse Architecture
  topic: keywords
subject: Data Sovereignty
  topic: keywords
subject: Big Data
  topic: keywords
subject: Data Governance
  topic: keywords
subject: Compliance
  topic: keywords
subject: Distributed Data Systems
  topic: keywords
subject: Multi-Cloud
  topic: keywords
subject: Data Pipelines
  topic: keywords
subject: Data Security
  topic: keywords
subject: Data Warehousing
  topic: keywords
subject: Regulatory Compliance
  topic: keywords
subject: Cloud Data Platforms
  topic: keywords
creator: Seshagiri N
rights: Copyright (c) 2023
date: 2023-05-09
publication date: 2023-05-09
starting page: 01
ending page: 10
doi: 10.67228/30715717/IJDEIC-2023PI7V1F

Integration of DataOps with Automated Exploratory Data Analysis (EDA) Workflows
type: item
description: In the age of data-driven innovation, rapid and reliable insight generation is critical. Exploratory Data Analysis (EDA) plays a foundational role in understanding data quality, structure, and relationships. However, traditional EDA processes are time-consuming and often disconnected from the operationalized data pipelines managed by DataOps. This paper explores the integration of DataOps principles with automated EDA workflows, aiming to enhance collaboration, reproducibility, and scalability in modern data ecosystems. By aligning automation tools with robust CI/CD pipelines, teams can streamline data profiling, visualization, and anomaly detection in real time. We propose an architectural framework for this integration, illustrate a practical use case, and discuss the implications for both data engineering and data science teams. This convergence promises a more agile and intelligent approach to data exploration, ensuring faster time-to-insight and improved data trustworthiness.
subject: Articles
  topic: section
subject: DataOps
  topic: keywords
subject: Automated EDA
  topic: keywords
subject: Exploratory Data Analysis
  topic: keywords
subject: Data Pipelines
  topic: keywords
subject: CI/CD in Data Science
  topic: keywords
subject: Data Engineering
  topic: keywords
subject: AutoEDA Tools
  topic: keywords
subject: Data Automation
  topic: keywords
subject: Machine Learning Operations
  topic: keywords
subject: Data Quality
  topic: keywords
creator: Dana Scott
creator: John Cocke
rights: Copyright (c) 2023
date: 2023-04-12
publication date: 2023-04-12
starting page: 01
ending page: 10
doi: 10.67228/30715717/IJDEIC-2023PI5R8W

Metadata-Centric Semantic Layer Management in Multi-Domain Pipelines
type: item
description: In modern data-driven ecosystems, multi-domain data pipelines present significant challenges in terms of interoperability, scalability, governance, and semantic consistency. Traditional data integration methods often falter when faced with the heterogeneity and velocity of domain-specific data. This paper proposes a metadata-centric approach to managing the semantic layer in multi-domain data pipelines. By elevating metadata to a first-class citizen, organizations can dynamically model, interpret, and govern data semantics across domains without excessive data movement or manual schema alignment. We introduce a reference architecture and management framework that leverages active metadata, semantic annotations, knowledge graphs, and policy-driven governance to enable scalable, federated analytics. Case studies and experimental evaluation demonstrate the effectiveness of the approach in improving query performance, data lineage traceability, and cross-domain interoperability. The proposed methodology aligns with modern data mesh principles and promotes sustainable data infrastructure design.
subject: Articles
  topic: section
subject: Multi-Domain Pipelines
  topic: keywords
subject: Data Governance
  topic: keywords
subject: Data Mesh
  topic: keywords
subject: Knowledge Graphs
  topic: keywords
subject: Active Metadata
  topic: keywords
subject: Data Lineage
  topic: keywords
subject: Semantic Interoperability
  topic: keywords
subject: Federated Data Architecture
  topic: keywords
creator: Kathleen Booth
rights: Copyright (c) 2023
date: 2023-03-10
publication date: 2023-03-10
starting page: 01
ending page: 09
doi: 10.67228/30715717/IJDEIC-2023PI4K3Q

Industrial Big Data Analytics: Real-Time Decision-Making in Manufacturing
type: item
description: The manufacturing sector is undergoing a paradigm shift with the integration of Industry 4.0 technologies, particularly Industrial Big Data Analytics (BDA), which leverages high-velocity data from IoT sensors, PLCs, and production systems to enable real-time decision-making. This paper presents a novel edge-cloud analytics framework designed to address critical challenges in modern manufacturing, including data heterogeneity, latency bottlenecks, and cybersecurity risks. By implementing a hybrid architecture, the system processes sensor data at the edge (e.g., vibration spectra, thermal images) with <50ms latency for time-sensitive tasks like defect detection, while cloud-based machine learning models (e.g., LSTMs) perform long-term predictive maintenance with 89% accuracy. A large-scale case study conducted at an automotive assembly line demonstrated a 20% increase in production throughput and 15% reduction in unplanned downtime, translating to $2.7M annual cost savings. Key innovations include: (1) a dynamic data normalization pipeline (Eq. 1) that handles skewed industrial datasets; (2) a comparative analysis of ML models, showing Random Forest outperforms ANN/SVM in defect classification (92.4% F1-score); and (3) a priority-based edge processing system that reduces cloud bandwidth usage by 60%. Despite these advancements, the study identifies persistent hurdles such as legacy system interoperability (resolved via OPC UA gateways) and adversarial robustness in edge ML models. The paper concludes with a roadmap for future work, including federated learning for multi-plant scalability and digital twin integration for simulation-driven analytics. These findings validate BDA as a transformative tool for smart manufacturing, offering a 5.2-month ROI and actionable insights for practitioners adopting Industry 4.0 solutions.
subject: Articles
  topic: section
subject: Big Data Analytics
  topic: keywords
subject: Industry 4.0
  topic: keywords
subject: Real-Time Decision-Making
  topic: keywords
subject: Predictive Maintenance
  topic: keywords
subject: IoT
  topic: keywords
subject: Machine Learning
  topic: keywords
subject: Smart Manufacturing
  topic: keywords
creator: Jacques Arsac
creator: Gérard Huet
rights: Copyright (c) 2023
date: 2023-02-13
publication date: 2023-02-13
starting page: 01
ending page: 09
doi: 10.67228/30715717/IJDEIC-2023PI2C4K

Impact of Data Versioning on Longitudinal Analytical Model Performance
type: item
description: In data-driven environments where datasets evolve over time, the challenge of maintaining consistent and high-performing analytical models becomes increasingly critical. This paper investigates the impact of data versioning on the performance of longitudinal analytical models. We explore how changes in data over time, captured through systematic versioning, influence model accuracy, stability, and generalizability. Using real-world longitudinal datasets and a comparative modeling framework, we assess various data versioning strategies and their effects on predictive performance. Our findings reveal that integrating data versioning not only enhances reproducibility but also enables more robust handling of performance drift over time. This research offers practical insights for data scientists and engineers aiming to maintain the fidelity of analytical systems in dynamic environments.
subject: Articles
  topic: section
subject: Data Versioning
  topic: keywords
subject: Longitudinal Analysis
  topic: keywords
subject: Model Performance Drift
  topic: keywords
subject: Temporal Data
  topic: keywords
subject: Reproducibility
  topic: keywords
subject: Predictive Modeling
  topic: keywords
subject: Data Management
  topic: keywords
subject: Time-Series
  topic: keywords
subject: Machine Learning
  topic: keywords
subject: Data Lineage
  topic: keywords
creator: Heinz Zemanek
rights: Copyright (c) 2022
date: 2022-12-07
publication date: 2022-12-07
starting page: 01
ending page: 09
doi: 10.67228/30715717/IJDEIC-2022PII8M5P

Hybrid Data Models for Industrial Internet of Things (IIoT) Analytics
type: item
description: The Industrial Internet of Things (IIoT) has ushered in a new era of connected devices, real-time data collection, and intelligent analytics. However, the complexity and heterogeneity of IIoT environments demand sophisticated data modeling techniques to harness the full potential of analytics. Hybrid data models—integrating structured, semi-structured, and unstructured data representations—are emerging as a critical solution. This paper presents a comprehensive study on the design and application of hybrid data models tailored for IIoT analytics. It explores data heterogeneity challenges, presents layered data model architectures, and discusses the integration of relational, NoSQL, and time-series databases. A proposed hybrid data model architecture is introduced, with a focus on real-time processing, scalability, and semantic interoperability. The model is validated using an industrial case study involving predictive maintenance in a smart factory. Results indicate a significant improvement in data query performance, storage efficiency, and analytical accuracy compared to traditional single-model approaches. Flowcharts and diagrams illustrate data flow, architectural layers, and analytics pipelines. The paper concludes by emphasizing the role of hybrid data models as enablers for scalable, resilient, and intelligent IIoT ecosystems.
subject: Articles
  topic: section
subject: Industrial Internet of Things (IIoT)
  topic: keywords
subject: Hybrid Data Models
  topic: keywords
subject: Predictive Analytics
  topic: keywords
subject: NoSQL
  topic: keywords
subject: Time-Series Data
  topic: keywords
subject: Data Integration
  topic: keywords
subject: Edge Computing
  topic: keywords
subject: Semantic Interoperability
  topic: keywords
subject: Big Data
  topic: keywords
subject: Smart Manufacturing
  topic: keywords
creator: Howard Aiken
creator: Presper Eckert J
creator: John W. Mauchly
rights: Copyright (c) 2022
date: 2022-11-09
publication date: 2022-11-09
starting page: 01
ending page: 14
doi: 10.67228/30715717/IJDEIC-2022PII6B2R