Bigdata Ecosystem

I have had a lot of discussions on big data with my clients and prospects.  During these discussions some questions comes up on Hadoop for ex: – What are the different components of a Hadoop ecosystem?

In my point of view the question should be - What are different components of a Bigdata ecosystem?

Everyone seems to have a different answer for it.  I have tried to consolidate the answers. The results are this picture.

Book Review: Learning Spark: Lightning-Fast Big Data Analytics

Book Review:  Learning Spark: Lightning-Fast Big Data Analytics by Holden Karau, Andy Konwinski, and Matei Zaharia: Publisher- O'Reilly: ISBN- 13: 978-1449358624
 

Learning Spark: Lightning – Fast Big Data Analytics is still in Early Release phase and will be available in Feb 2015.  I have reviewed first seven chapters of the book which are still raw but coming neat and clean.

This book is very good introduction for newbies to Spark which is rage in Big Data domain. Book almost all samples in three languages – Java, Scala and Python which makes easier for lots of people to try them out and learn Spark.

The first chapter is just gives introduction while second chapter onward real fun starts. Chapter 2 lets you to install Spark on your laptop.  Chapters 3 to 6 talk about programming aspects of Spark. Chapter 7 is about Spark cluster.

I am expecting book will be good one.

Disclaimer: I did not get paid to review this book, and I do not stand to gain anything if you buy the book. I have no relationship with the publisher or the author. I got electronic format of book from publisher for review.

One can get more information about book and related topics from:

1. Amazon: http://www.amazon.com/Learning-Spark-Lightning-Fast-Data-Analytics/dp/1449358624
2. Publisher -- Oreilly http://shop.oreilly.com/product/0636920028512.do

Book Review: Using Flume: Stream Data into HDFS and HBase

Book Review:  Using Flume: Stream Data into HDFS and HBase by Hari Shreedharan: Publisher- O'Reilly: ISBN- 13: 978-1449368302

Using Flume: Stream Data into HDFS and HBase is for developers as well as Administrators of Hadoop clusters.  In its first chapter book discusses HBase which is little puzzling but as book progresses, it takes you for  deep dive in various aspects of Flume.  Book covers Streaming of data, various sources, channels, sinks, interceptors, and other components of Flume.

The last chapter is about administration of Flume which is very short. This chapter might be little bit in depth to cover capacity planning, deployment options, etc.

Nevertheless, book is a good reference for any person playing in Hadoop playground.

Disclaimer: I did not get paid to review this book, and I do not stand to gain anything if you buy the book. I have no relationship with the publisher or the author. I got electronic format of book from publisher for review.

Further reading: Apache Flume: Distributed Log Collection for Hadoop (http://www.amazon.com/Apache-Flume-Distributed-Collection-Hadoop/dp/1782167919)

One can get more information about book and related topics from:

1. Amazon: http://www.amazon.com/Using-Flume-Stream-Data-HBase/dp/1449368301
2. Publisher -- Oreilly http://shop.oreilly.com/product/0636920030348.do

Book Review: Hadoop Operations

Book Review:  Hadoop Operations by Eric Sammer: Publisher- O'Reilly: ISBN- 13: 978-1449327057




Hadoop Operations by Eric Sammer is marvelous book which explains almost each bit of information is very lucid manner. AS name suggests, book is for operations guys - How data is ingested and replicated, or how MapReduce "finds" the most suitable
node to run parts of job, or what the cost and performance advantages are of adopting the shared-nothing, commodity hardware model recommended for Hadoop cluster, etc.

 

This book is for Operations guys/Administrators and as good supporting material to developers.

 

 

Disclaimer: I did not get paid to review this book, and I do not stand to gain anything if you buy the book. I have no relationship with the publisher or the author. I got electronic format of book from publisher for review.

Further reading: There are several books on similar topic. Hadoop Operations and ClusterManagement Cookbook and Hadoop Cluster Deployment.

One can get more information about book and related topics from:

 

1. Amazon: http://www.amazon.com/Hadoop-Operations-Eric-Sammer/dp/1449327052
2. Publisher -- Oreilly http://shop.oreilly.com/product/0636920025085.do

 

Sizing of Name Node Ram and Physical Memory for Data Nodes

Recently while working with one of client, I was asked to advice about RAM requirement for Name Node and Physical storage capacity for Data Nodes. This is one of the questions, I am asked repeatedly. To solve the issue once for all, I like to formalize the answer in terms of mathematical formula, so ambiguity can be take out from answer.

 

 

The associated Excel file is also created and available at scribd.

Hadoop NameNode RAM and Physical Memory for DataNodesSizing