OPTIMIZED TASK PARALLELISM FOR BIG DATA ANALYTICS IN A DISTRIBUTED SYSTEM

Abstract

As the scale of big data grows, efficiently processing and analyzing such vast amounts of information becomes increasingly challenging. Descriptive statistical analysis, crucial in many domains, often demands significant computation time when applied to large datasets. This research addresses the problem of slow execution in performing descriptive statistical analysis and generating graphs on large datasets, which is a bottleneck for timely insights and decision-making. The objective of this study is to optimize this process by leveraging task parallelism in a high-performance computing (hpc) environment. An optimized task-parallelism is introduced, which distributes computational tasks across multiple nodes using the message passing interface (mpi). Specifically, tasks such as calculating statistics and generating visual plots (histograms, percentiles, qq plots, cumulative distribution functions, etc.) Are divided among nodes in a distributed architecture, significantly reducing the overall processing time. This approach was evaluated using a large taxi trip dataset comprising approximately two million records, comparing the execution time of sequential and parallel processing methods. The results demonstrate that the parallel implementation achieves a 71.29% reduction in execution time, decreasing from 246.83 seconds (sequential) to 70.89 seconds (parallel). This significant improvement highlights the efficiency and scalability of the proposed task-parallel solution, offering valuable insights into enhancing big data analysis performance in distributed environments.