CloudBLAST: Combining MapReduce and Virtualization on Distributed Resources for Bioinformatics Applications

TitleCloudBLAST: Combining MapReduce and Virtualization on Distributed Resources for Bioinformatics Applications
Publication TypeConference Paper
Year of Publication2008
AuthorsMatsunaga, A, Tsugawa, M, Fortes, J
Conference Name2008 IEEE Fourth International Conference on eScience (eScience)
Conference LocationIndianapolis, IN, USA
ISBN Number978-1-4244-3380-3
Keywordsapplication program interfaces, Application virtualization, bioinformatics, bioinformatics applications, bioinformatics tool, cloud computing, CloudBLAST, Computer network management, computing resources, Distributed computing, distributed resources, Environmental management, MapReduce, message passing, message passing interface, MPI-based solution, network virtualization, Resource management, Resource virtualization, Technology management, Testing, virtual machines, Virtual machining, virtual network technology, virtualization, WAN-based test bed, wide area networks
AbstractThis paper proposes and evaluates an approach to the parallelization, deployment and management of bioinformatics applications that integrates several emerging technologies for distributed computing. The proposed approach uses the MapReduce paradigm to parallelize tools and manage their execution, machine virtualization to encapsulate their execution environments and commonly used data sets into flexibly deployable virtual machines, and network virtualization to connect resources behind firewalls/NATs while preserving the necessary performance and the communication environment. An implementation of this approach is described and used to demonstrate and evaluate the proposed approach. The implementation integrates Hadoop, Virtual Workspaces, and ViNe as the MapReduce, virtual machine and virtual network technologies, respectively, to deploy the commonly used bioinformatics tool NCBI BLAST on a WAN-based test bed consisting of clusters at two distinct locations, the University of Florida and the University of Chicago. This WAN-based implementation, called CloudBLAST, was evaluated against both non-virtualized and LAN-based implementations in order to assess the overheads of machine and network virtualization, which were shown to be insignificant. To compare the proposed approach against an MPI-based solution, CloudBLAST performance was experimentally contrasted against the publicly available mpiBLAST on the same WAN-based test bed. Both versions demonstrated performance gains as the number of available processors increased, with CloudBLAST delivering speedups of 57 against 52.4 of MPI version, when 64 processors on 2 sites were used. The results encourage the use of the proposed approach for the execution of large-scale bioinformatics applications on emerging distributed environments that provide access to computing resources as a service.