apply for a AWS acct

• go to http://aws.amazon.com/account/
• put name, email, click on "sign in";
• put full name, email, billing info, credit bard;
• pass the phone number authorization;
• select AWS support plan, use Basic (Free);
• see message "Thank you for updating your Amazon Web Services Account!”.

log in your acct

• sign in using the acct/pass you just made.
• choose "Launch the AWS Management Console";
• click on button "sign in to the AWS console";
• choose "I am a returning user and my password is";
• click "sign in using our secure server";
• see "EC2 Dashboard" on the top left corner.

create your key pair

• you authorize yourself and communicate with Amazon cloud using public and private key pair.
• on your left panel of EC2 dashboard, click on "network & security" category's "Key Pairs";
• click on button "create key pair";
• type your key pair name, for me, I put "EC2UbuntuLTSThreeT1Micro";
• see a download file window, choose a secure folder to save your .pem file which will be your private key. For me, I get the file called "EC2UbuntuLTSThreeT1Micro.pem";

create EC2 instances

• How to understand instance? each instance is like a seperate machine(or pc or laptop).
• on your left panel of EC2 dashboard,, click on "instances" category's "instances";
• click on button "launch instance"; choose the system, find the 9th options--"Ubuntu Server 12.04 LTS (HVM) - ami-5fa4a236 64 bit";
• click on "select";
• keep the default setting, choose "next:configure instance details";
• put "3" into "Number of instances";
• click on "review and launch";
• click on "launch";
• choose the key pair "EC2UbuntuLTSThreeT1Micro";
• check the acknowleage box;
• click on "launch instance",you should see "Your instances are now launching"; scroll down to the bottom, click on "view instances",click one each instance and rename them, such as "meetup1","meetup2","meetup3",wait till you see the status checks is changed from "initializing" to "2/2 checks passed".

configure security group

• each group is like a firewall. The nodes of the same cluster need to be in the same security group.
• click on "create security groups",put name such as "launch-wizard-1" or change to your customized name;
• under "inbound" tab, cick on add rules

  • add five rules:

    • choose type="SSH", "save";
    • choose type="ALL ICM", Source=Anywhere, "save";
    • choose type="Custom TCP", port range="9000", choose "anywhere",save;
    • choose type="Custom TCP", port range="9001", choose "anywhere",save;
    • choose type="Custom TCP", port range="50000 - 50100", choose "anywhere", save;

    double check all your inbound setting, they should look like:
    
    | Type            | Protocol  | Port Range      | Source            |
    | ----------------|:---------:| ---------------:|------------------:|
    | SSH             | TCP       | 22              |Anywhere 0.0.0.0/0 |
    | Custom TCP Rule | TCP       | 9001            |Anywhere 0.0.0.0/0 |
    | Custom TCP Rule | TCP       | 9000            |Anywhere 0.0.0.0/0 |
    | Custom TCP Rule | TCP       | 50000 - 50100   |Anywhere 0.0.0.0/0 |
    | All ICMP        | ICMP      | 0 - 65535(or na)|Anywhere 0.0.0.0/0 |
    
    

manage your instance

• know how to turn on and off the instances
• select the instance, right click to choose "stop", you won't be able to use this instance and won't be charged;
• select the instance, right click to choose "start", you can use the instance and will be charged. --next time, if you restart the instance, your "public DNS" will be different, but your "private DNS" will not changed. If you reboot the instance, your "public DNS" will be the same as before rebooting.

You are required to use vi editor. The basic operations are:

 Cheet sheet for vi

| what you want to do  | Key stokes       | 
| ---------------------|:----------------:| 
| insert/edit model    | i                | 
| finish and save      | ESC->:->wq->Enter|
| finish and not save  | ESC->q!->Enter   |
| go to the end of line| ESC->o           |

generate your server rsa key for three instances

download haoop source codes

• wget http://mirror.cc.columbia.edu/pub/software/apache/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz
• tar -xzvf hadoop-1.2.1.tar.gz
• cd hadoop-1.2.1/conf

configure the environment

• configure Java Path
  • vi hadoop-env.sh
  • search for the line "# The java implementation to use. Required."
  • delete the "#"
  • set "export JAVA_HOME=/usr/lib/jvm/java-7-oracle"
  • save the file
• configure core-site file -vi core-site.xml

  • between and , put fs.default.name hdfs://meetup1:9000

    hadoop.tmp.dir /home/ubuntu/hadoop/tmp
• make new folder "~/hadoop/tmp"
  • cd ~
  • mkdir hadoop
  • cd hadoop
  • mkfir tmp
• configure redundance, the value is usually set as 1 or 2 and less than total number of slave nodes.
  • cd ~
  • cd hadoop-1.2.1/conf
  • vi hdfs-site.xml
  • between and , put dfs.replication 2
• configure master file
  • delect localhost
  • put meetup1
• configre slave file
  • delect localhost
  • put meetup2 meetup3
• copy all the configuration from master node to two slave nodes
  • scp -r ~/hadoop-1.2.1 meetup2:/home/ubuntu
  • scp -r ~/hadoop-1.2.1 meetup3:/home/ubuntu

format

• ~/hadoop-1.2.1/bin/hadoop namenode -format
• you should get "Storage directory /home/ubuntu/hadoop/tmp/dfs/name has been successfully formatted."
• cd ~/hadoop/tmp/
• you should find the two folders
  • dfs->name->current
  • dfs->name->image

start your hadoop