Map Reduce: For my sophomore

Yash: Dad, What is Map Reduce?

Me: Ok!! Give me context.

Yash: Alex was talking about it and saying, Google invented this thing. What is Map Reduce? Is it something to do with Google Glasses?

Me: It has nothing to do with Google Glass. It is an algorithm to do lot of work in short duration using parallel processing.

GJ: Boys food is ready. Come to dinner table.

Me: Let’s eat food and we will talk there.

At dinner table.

Yash: So what is Map Reduce?

Me: Ok let us talk about food first. Tell me how she prepares a curry.

Yash: For every curry there is separate recipe. She follows that recipe.

Me: hmm…

Me: But there should be some very high level steps, she must be following in each recipe.

Yash: Oh yes. She follows few steps in each preparation.

• Wash the vegetables
• Chop the vegetables
• Prepares masala (combination of spices)
• Put everything into pan and then fry and simmer.

Me: Very good. But if she needs to cook food for 50 persons, how she will do?

Yash: She will take your and mine help. I can wash the vegetables. You can do chop job. She can prepare masala. If we all three work parallel, she can cook in less time.

Me: Excellent. Now she has to cook curry for 300 persons. What to do?

Yash: We need to hire help to do the job.

Me: How?

Yash: Say we hire 12 people. First 3 hires can start washing vegetables.  Once we have sufficient vegetables to start chopping job, next 4 hires will start chopping job. In parallel another 3 hires can start preparing masala. Last 2 hires can combine masala and chopped vegetables and start frying and simmering.

Me: Fantastic!! Now you know map Reduce.

Yash: I didn’t get it.

Me: Let us put preparing curry in Map Reduce context. As you explained first three batches of hires will be doing parallel work of washing, chopping and masala preparation. This is Map, where lot of parallel work is happening. Last batch of hires will combine the output of previous batches and do the frying and simmering job. This is Reduce part. Here work from Map part are combined and fed into another process.

Give a paper napkin. Let me put this into picture.

Yash: It has nothing to do with Google Glass.

Me: Yes. Map Reduce has nothing to do with Google Glass or ant programming language. You can implement Map reduce in any programming language. Even you can implement Map Reduce to situations where you have nothing to do with computers.

Architecture and Agile

Often claimed by Architects, sustainable Architecture is first causality of Agile. How truthful is this statement?

Let us examine arguments from both sides.

Agile enthusiastic

• Agile is about working software rather than about some imaginary nonfunctional requirements (read scalability, maintainability, etc.)
•  As one of the famous German military leader had said - No battle plan survives contact with the enemy. In simple words, practicalities mean that the exact letter of the guidelines cannot always be followed so big upfront abstract architecture is waste of resources.

Waterfall enthusiastic

• On the fly architecture (not the code) leads to spaghetti architecture which results in increased maintenance cost.

What is the truth?

I say truth often lies in middle not on either end.   

Often Agile purists quote Lean and JIT and demand architecture on the fly. Is comparison fair? Are we comparing apple to apple?  Lean and JIT have roots in manufacturing.  Let us take JIT for Car Assembly line. JIT is practiced while assembling car not while building assembly line and certainly not when car Assembly line is architected and designed.  And I just remember that there is still not any assembly line for software development even after repeated attempts by various heavy weights.

I see software architecture as multi stage. Even I like to add solutioning as founding stone of software architecture.  If one compares making a painting with software, that argument, I will buy. Each painting is different from other. Before start to paint, painter has a vision, what he wants to paint – A portrait or scenery or something else. Similarly A solutioning guy knows what he wants as end product. Painter first sketch outline and then slowly fill in the space with colors.  

 

Does software architecture follow same pattern? I think so.  In starting, painter has vision and blurred idea of final picture but as construction steps progress, painter keep on adding shades and brightness to achieve final painting.  On similar lines, software architect must have vision of what he is aiming for and as development progresses, with help of evolving business needs and under constraints of past acts (or misdeeds) keep on manipulating architecture.

Planning Line

While talking to Agile enthusiastic, often discussion venture into waterfall.  Now the fun starts. Each side becomes religiously adherent to their position and none want to understand other’s point of view.  Is there any common point?

I think, similar to number line, if we think of Planning Line. One end of planning line ends (!) at CHAOS and other at IMMOTILE. Now on this line let’s plot different methodologies.

What do you think of position of these methodologies on Planning Line?