Hadoop入门-WordCount示例_光环大数据培训

光环大数据光环大数据-大数据培训知名品牌大数据培训知名品牌http:/hadoop.aura.cn 光环大数据光环大数据 http:/hadoop.aura.cnHadoopHadoop 入门入门-WordCount-WordCount 示例示例_ _光环大数据培训光环大数据培训光环大数据培训光环大数据培训，WordCount 的过程如图，这里记录下入门的过程，虽然有很多地方理解的只是皮毛。hadoop 的安装安装比较简单，安装完成后进行单机环境的配置。hadoop-env.sh:指定 JAVA_HOME。# The only required environment variable is JAVA_HOME. All others are# optional. When running a distributed configuration it is best to# set JAVA_HOME in this file, so that it is correctly defined on# remote nodes.# The java implementation to use.export JAVA_HOME=“$(/usr/libexec/java_home)“core-site.xml：设置 Hadoop 使用的临时目录，NameNode 的地址。hadoop.tmp.dir /usr/local/Cellar/hadoop/hdfs/tmp fs.default.name hdfs:/localhost:9000 hdfs-site.xml：一个节点，副本个数设为 1。dfs.replication 1 光环大数据光环大数据-大数据培训知名品牌大数据培训知名品牌http:/hadoop.aura.cn 光环大数据光环大数据 http:/hadoop.aura.cnmapred-site.xml:指定 JobTracker 的地址。mapred.job.tracker localhost:9010 启动 Hadoop 相关的所有进程。 sbin git:(master) ./start-all.shThis script is Deprecated. Instead use start-dfs.sh and start-yarn.sh16/12/03 19:32:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform. using builtin-java classes where applicableStarting namenodes on localhostPassword:localhost: starting namenode, logging to /usr/local/Cellar/hadoop/2.7.1/libexec/logs/hadoop-vonzhou-namenode-vonzhoudeMacBook-Pro.local.outPassword:localhost: starting datanode, logging to /usr/local/Cellar/hadoop/2.7.1/libexec/logs/hadoop-vonzhou-datanode-vonzhoudeMacBook-Pro.local.outStarting secondary namenodes 0.0.0.0Password:0.0.0.0: starting secondarynamenode, logging to /usr/local/Cellar/hadoop/2.7.1/libexec/logs/hadoop-vonzhou-secondarynamenode-vonzhoudeMacBook-Pro.local.out16/12/03 19:33:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform. using builtin-java classes where applicablestarting yarn daemonsstarting resourcemanager, logging to /usr/local/Cellar/hadoop/2.7.1/libexec/logs/yarn-vonzhou-resourcemanager-vonzhoudeMacBook-Pro.local.outPassword:localhost: starting nodemanager, logging to /usr/local/Cellar/hadoop/2.7.1/libexec/logs/yarn-vonzhou-nodemanager-vonzhoudeMacBook-Pro.local.out（可以配置 ssh 无密码登录方式，否则启动 hadoop 的时候总是要密码。 ）看看启动了哪些组件。光环大数据光环大数据-大数据培训知名品牌大数据培训知名品牌http:/hadoop.aura.cn 光环大数据光环大数据 http:/hadoop.aura.cn sbin git:(master) jps -l5713 org.apache.hadoop.hdfs.server.namenode.NameNode6145 org.apache.hadoop.yarn.server.nodemanager.NodeManager6044 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager5806 org.apache.hadoop.hdfs.server.datanode.DataNode5918 org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode访问 http:/ localhost:50070/ 可以看到 DFS 的一些状态。WordCount 单词计数WordCount 就是 Hadoop 学习的 hello world，代码如下：public class WordCount public static class Map extends Mapper private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException String line = value.toString(); StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasMoreTokens() word.set(tokenizer.nextToken(); context.write(word, one); public static class Reduce extends Reducer public void reduce(Text key, Iterable values, Context context) throws IOException, InterruptedException int sum = 0; for (IntWritable val : values) sum += val.get(); context.write(key, new IntWritable(sum); public static void main(String args) throws Exception Configuration conf = new Configuration(); Job job = new Job(conf, “wordcount“); job.setJarByClass(WordCount.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); job.setMapperClass(Map.class); job.setReducerClass(Reduce.class); /* 光环大数据光环大数据-大数据培训知名品牌大数据培训知名品牌http:/hadoop.aura.cn 光环大数据光环大数据 http:/hadoop.aura.cn* 设置一个本地 combine,可以极大的消除本节点重复单词的计数,减小网络传输的开销 */ job.setCombinerClass(Reduce.class); job.setInputFormatClass(TextInputFormat.class); job.setOutputFormatClass(TextOutputFormat.class); FileInputFormat.addInputPath(job, new Path(args0); FileOutputFormat.setOutputPath(job, new Path(args1); job.waitForCompletion(true); 构造两个文本文件, 把本地的两个文件拷贝到 HDFS 中： hadoop-examples git:(master) ln /usr/local/Cellar/hadoop/2.7.1/bin/hadoop hadoop hadoop-examples git:(master) ./hadoop dfs -put wordcount-input/file* inputDEPRECATED: Use of this script to execute hdfs command is deprecated.Inst