- What is Big Data?
- Why all industries are talking about Big Data?
- What are the issues in Big Data?
- Storage & Processing
- What are the challenges for storing big data?
- What are the challenges for processing big data? What are the technologies support big data?
- Hadoop – Bigdata
- Traditional Databases Vs NOSQL
- Most popular BigData ecosystems (Spark, cassandhra, flink )
- Installation of apache Hadoop
- What is Hadoop?
- History of Hadoop
- Why Hadoop?
- Hadoop Use cases
- Advantages and disadvantages of Hadoop
- Importance of Different Ecosystems of Hadoop
- Importance of Integration with other BigData solutions Big Data Real time Use Cases
- Apache Hadoop installation in Local mode (hands on installation on ur laptop)
- Psuedo mode (hands on installation on ur laptop)
- Cluster mode ( 5 node cluster setup in aws Account)
- Importance of each command
- How to execute the command
- Hdfs admin related commands explanation
- Can we change the existing configurations of hdfs or not?
- CLI(Command Line Interface) using hdfs commands
- Name Node
- Importance of Name Node
- What are the roles of Name Node
- What are the drawbacks in Name Node
- Secondary Name Node
- Importance of Secondary Name Node
- What are the roles of Secondary Name Node
- What are the drawbacks in Secondary Name Node
- Data Node
- Importance of Data Node
- What are the roles of Data Node
- What are the drawbacks in Data Node
Data Storage in HDFS
- Traditional OS Block information.
- How blocks are storing in DataNodes
- How replication works in Data Nodes
- HDFS Block size
- Importance of HDFS Block size
- Why Block size is so large?
- How it is related to MapReduce split size
- Importance of HDFS Replication factor in production environment
- Can we change the replication for a particular file or folder
- Can we change the replication for all files or folders
- How to write the files in HDFS
- How to read the files in HDFS
- Rack Awareness, Topology Script
- How block replicated?
How to overcome the Drawbacks in HDFS
- Name Node failures
- Secondary Name Node failures
- Data Node failures
Where does it fit and Where doesn’t fit? Exploring the Apache HDFS Web UI
How to configure the Hadoop Cluster
- How to add the new nodes ( Commissioning )
- How to remove the existing nodes ( De-Commissioning )
- How to verify the Dead Nodes
- How to start the Dead Nodes
Map Reduce architecture
- Importance of JobTracker
- What are the roles of JobTracker
- What are the drawbacks in JobTracker
- Importance of TaskTracker
- What are the roles of TaskTracker
- What are the drawbacks in TaskTracker
- Map Reduce Job execution flow
Data Types in Hadoop
- What are the Data types in Map Reduce
- Text Input Format
- Key Value Text Input Format
- Sequence File Input Format
- Nline Input Format
- Importance of Input Format in Map Reduce
- How to use Input Format in Map Reduce
- How to write custom Input Format’s and its Record Readers Output Format’s in Map Reduce
- Text Output Format
- Sequence File Output Format
- Importance of Output Format in Map Reduce
- How to use Output Format in Map Reduce
- How to write custom Output Format’s and its Record Writers Mapper
- What is mapper in Map Reduce Job
- Why we need mapper?
- What are the Advantages and Disadvantages of mapper
- Writing mapper programs Reducer
- What is reducer in Map Reduce Job
- Why we need reducer ?
- What are the Advantages and Disadvantages of reducer
- Writing reducer programs Combiner
- What is combiner in Map Reduce Job
- Why we need combiner?
- What are the Advantages and Disadvantages of Combiner
Writing Combiner programs Partitioner
- What is Partitioner in Map Reduce Job
- Why we need Partitioner?
- What are the Advantages and Disadvantages of Partitioner
- Writing Partitioner programs Distributed Cache
- What is Distributed Cache in Map Reduce Job
- Importance of Distributed Cache in Map Reduce job
- What are the Advantages and Disadvantages of Distributed Cache
- Writing Distributed Cache programs Counters
- What is Counter in Map Reduce Job
- Why we need Counters in production environment?
- How to Write Counters in Map Reduce programs
- Importance of Writable and Writable Comparable Api’s
- How to write custom Map Reduce Keys using Writable
- How to write custom Map Reduce Values using Writable Comparable
- Map Side Join
- What is the importance of Map Side Join
- Where we are using it
- Reduce Side Join
- What is the importance of Reduce Side Join
- Where we are using it
- What is the difference between Map Side join and Reduce Side Join?
- Importance of Compression techniques in production environment
- Compression Types
- NONE, RECORD and BLOCK
- Compression Codecs
- Default, Gzip, Bzip, Snappy and LZO
- Enabling and Disabling these techniques for all the Jobs
- Enabling and Disabling these techniques for a particular Job
Map Reduce Schedulers
- FIFO Scheduler
- Capacity Scheduler
- Fair Scheduler
- Importance of Schedulers in production environment
- How to use Schedulers in production environment
- Map Reduce Programming Model
- How to write the Map Reduce jobs in Java
- Running the Map Reduce jobs in local mode
- Running the Map Reduce jobs in pseudo mode
- Running the Map Reduce jobs in cluster mode
Debugging Map Reduce Jobs
- How to debug Map Reduce Jobs in Local Mode.
- How to debug Map Reduce Jobs in Remote Mode.
- Hadoop 2.6 version features
- Introduction to Namenode fedoration
- Introduction to Namenode High Availabilty
- Difference between Hadoop 1.x.x and Hadoop 2.x.x versions
- HDFS changes in Hadoop 2.x
- Mapreduce Changes in 2.x
YARN (Next Generation Map Reduce)
- What is YARN?
- What is the importance of YARN?
- Where we can use the concept of YARN in Real Time
- What is difference between YARN and Map Reduce
- Data Locality
- What is Data Locality?
- Will Hadoop follows Data Locality?
- Speculative Execution
- What is Speculative Execution?
- Will Hadoop follows Speculative Execution?
Map Reduce Commands
- Importance of each command
- How to execute the command
- Mapreduce admin related commands explanation
- Can we change the existing configurations of mapreduce or not?
- Importance of configurations
- Power of Hadoop 2.x
- Writing Unit Tests for Map Reduce Jobs
- Configuring hadoop development environment using Eclipse
- Use of Secondary Sorting and how to solve using MapReduce
- How to Identify Performance Bottlenecks in MR jobs and tuning MR jobs. Map Reduce Streaming and Pipes with examples
- Exploring the Apache MapReduce Web UI
- Hive Introduction Hive architecture
- Semantic Analyzer
- Hive Integration with Hadoop
- Hive Query Language(Hive QL) VS SQL
- Hive Installation and Configuration Hive
- Map-Reduce and Local-Mode Hive DLL and DML Operations
- Hwi Metastore
- embedded metastore configuration
- external metastore configuration UDF’s
- How to write the UDF’s in Hive
- How to use the UDF’s in Hive
- Importance of UDF’s in Hive
- How to use the UDAF’s in Hive
- Importance of UDAF’s in Hive UDTF’s
- How to use the UDTF’s in Hive
- Importance of UDTF’s in Hive How to write a complex Hive queries What is Hive Data Model?
- Importance of Hive Partitions in production environment
- Limitations of Hive Partitions
- How to write Partitions
- Importance of Hive Buckets in production environment
- How to write Buckets
- Importance of Hive SerDe’s in production environment
- How to write SerDe programs
- How to integrate the Hive and Hbase
- Json Serde and Regex serde
- Introduction to Sqoop
- MySQL client and Server Installation
- Sqoop Installation
- How to connect to Relational Database using Sqoop
- Sqoop Commands and Examples on Import and Export commands
- Introduction to flume Flume installation
- Integrate flume with Hadoop
- Gather log data from twitter and pull into Flume
- Flume agent usage and Flume examples execution
- Apache OOZIE Introduction to oozie Oozie installation
- Executing oozie workflow jobs
- Monitering Oozie workflow jobs
- Introduction to Apache Pig
- Map Reduce Vs Apache Pig
- SQL Vs Apache Pig Different data types in Pig
- Modes Of Execution in Pig
- Local Mode
- Map Reduce Mode
- Execution Mechanism
- Grunt Shell
UDF’s in Pig
- How to write the UDF’s in Pig
- How to use the UDF’s in Pig
- Importance of UDF’s in Pig
- How to write the Filter’s in Pig
- How to use the Filter’s in Pig
- Importance of Filter’s in Pig
- How to write the Load Functions in Pig
- How to use the Load Functions in Pig
- Importance of Load Functions in Pig
- How to use the Store Functions in Pig
- Importance of Store Functions in Pig
- Transformations in Pig
- How to write the complex pig scripts
- How to integrate the Pig and Hbase
- Introduction to zookeeper
- Pseudo mode installations
- Zookeeper cluster installations
- Basic commands execution
- Zookeeper Architecture
- How it’s functioning with Hadoop
- Hbase introduction
- Hbase usecases Hbase basics
- Column families
- Hbase installation
- Local mode
- Psuedo mode
- Cluster mode
- Hbase Architecture
- WriteAhead Log
- Log Structured MergeTrees
- Mapreduce integration
- Mapreduce over Hbase
- Hbase Usage
- Key design
- Bloom Filters
- Hbase Clients
- Web Based UI
- Schema definition
- Basic CRUD operations
- Create an instance in AWS
- Setup a cluster in Aws
- Realtime practice Hadoop.
- Introduction to Amazon EMR and Amazon Ec2
- How to use Amazon EMR and Amazon Ec2
- Why to use Amazon EMR and Importance of this
- Introduction to Cloudera
- Cloudera Installation
- Cloudera Certification details How to use cloudera hadoop
- What are the main differences between Cloudera and Apache Hadoop
- Cloudera mock test for Certification
- Hortonworks Distribution Introduction to Hortonworks
- Hortonworks Installation Hortonworks Certification details
- How to use Hortonworks hadoop
- What are the main differences between Hortonworks and Apache Hadoop
Hadoop Administration tasks
- Ganglia, Nagios (Monitoring tools)
- How those monitoring tools work in real time?
- Hadoop Administrator main functionalities.
- Trouble shouting
Day 39: Kafka with Strom:
- Kafka installation.
- How kafka sends million messages.
- Power of Kafka.
- Importance in Strom.
Day 40: Future of Hadoop:
- What next after Hadoop?
- Apache Spark introduction
- Installation of Spark.
Pre-Requisites for this Course
- Java Basics like OOPS Concepts, Interfaces, Classes and Abstract
- SQL Basic Knowledge
- Linux Basic Commands
Every Saturday and Sunday holiday, but assists through online to implement POCs and rectify the problems.