WordCount in Spark Scala

posted in: hadoop | 0

Hello, In theses videos, I am explaining about how to install eclipse, how to install scala? how to create appropriate configurations in eclipse, maven to implement spark applications, finally how to run spark wordcount program in maven build tool.

How to install Eclipse in Ubuntu:

If you want this script just mail me at venu@bigdataanalyst.in i will mail

Download eclipse from eclipse
http://mirrors.ustc.edu.cn/eclipse/technology/epp/downloads/release/mars/R/eclipse-jee-mars-R-linux-gtk-x86_64.tar.gz
put somewhere where you want /home/hadoop/work
extract
tar -zxvf /home/hadoop/work/eclipse-jee-mars-R-linux-gtk-x86_64.tar.gz
gksudo gedit /usr/share/applications/eclipse.desktop
#enter password
#paste it
[Desktop Entry]Name=Eclipse 4
Type=Application
Exec=/home/hadoop/work/eclipse/eclipse
Terminal=false
Icon=/home/hadoop/work/eclipse/icon.xpm
Comment=Integrated Development Environment
NoDisplay=false
Categories=Development;IDE;
Name[en]=Eclipse

##how to install scala plugin in Eclipse#####
#First check updates and updates to prevent problems in future
go to Help>check for updates> next>next>accept conditions>finish // wait 5 min restart the eclipse
go to Help>eclipse marketplace>find-> scala> scala ide> confirm> next>next>accept>finish

after create project in maven, right click and go to configure >add scala nature

#####How to create a maven project and hellow world scala program####
for spark Streaming: http://mvnrepository.com/artifact/org.apache.spark/spark-streaming_2.10/1.6.0
spark core: http://mvnrepository.com/artifact/org.apache.spark/spark-core_2.10
spark sql: http://mvnrepository.com/artbifact/org.apache.spark/spark-sql_2.10
scala: http://mvnrepository.com/artifact/org.scala-lang/scala-library/2.10.6
Hadoop:http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common/2.7.2

#based on your spark , scala, hadoop version change it.

WordCount using Spark Scala