Please login to post comment

What is Spark?

  • Amruta Bhaskar
  • Dec 10, 2019
  • 0 comment(s)

Apache Spark is an open supply massive processing framework designed around speed, simple use, and complicated analytics. It had been originally developed in 2009 in UC Berkeley’s AMPLab, and open-sourced in 2010 as an Apache project.


Spark has many blessings compared to different massive information and MapReduce technologies like Hadoop and Storm. To begin with, Spark provides America with a comprehensive, unified framework to manage massive processing needs with a spread of knowledge sets that are numerous in nature (text data, graph data etc) similarly because of the supply of knowledge (batch v. period of time streaming information).


Spark allows applications in Hadoop clusters to run up to a hundred times quicker in memory and ten times faster even once running on disk.


Spark permits you to quickly write applications in Java, Scala, or Python. It comes with an intrinsic set of over eighty high-level operators. And you'll be able to use it interactively to question information at intervals the shell.


In addition to Map and scale back operations, it supports SQL queries, streaming data, machine learning and graph processing. Developers will use these capabilities complete or mix them to run in a very single information pipeline use case.

In this initial instalment of Apache Spark article series, we'll investigate what Spark is, how it compares with a typical MapReduce resolution and the way it provides an entire suite of tools for giant processing.


Hadoop and Spark

Hadoop as an enormous processing technology has been around for ten years and has evidenced to be the answer of selection for processing massive information sets. MapReduce could be a nice resolution for one-pass computations, however, not terribly economical to be used cases that need multi-pass computations and algorithms. Every step within the processing advancement has one Map section and one cut back phase and you'll have to be compelled to convert any use case into MapReduce pattern to leverage this resolution.


The Job output information between every step must be held on within the distributed classification system before the subsequent step will begin. Hence, this approach tends to slow because of replication and disk storage. Also, Hadoop solutions usually embrace clusters that are laborious to line up and manage. It conjointly needs the mixing of many tools for various huge information use cases (like a driver for Machine Learning and Storm for streaming data processing).


If you wished to try to to one thing difficult, you will have to string along a series of MapReduce jobs and execute them in sequence. Each of these jobs was high-latency, and none might begin until the previous job had finished fully.


Spark permits programmers to develop advanced, multi-step information pipelines mistreatment directed acyclic graph (DAG) pattern. It conjointly supports in-memory information sharing across DAGs, so totally different jobs will work with constant information.


Spark runs on prime of existing Hadoop Distributed File System (HDFS) infrastructure to supply increased and extra practicality. It provides support for deploying Spark applications in Associate in Nursing existing Hadoop v1 cluster (with SIMR – Spark-Inside-MapReduce) or Hadoop v2 YARN cluster or maybe Apache Mesos.


We should investigate Spark as an alternative to Hadoop MapReduce instead of a replacement to Hadoop. It’s not supposed to exchange Hadoop. However, to supply a comprehensive and unified resolution to manage totally different huge information use cases and necessities.


Author- Chethan M

Please login to post comment

( 0 ) comment(s)