求助一个 MapReduce 问题，关于 Mapper 的 Type 不正确问题

V2EX = way to explore

V2EX 是一个关于分享和探索的地方

For Existing Member Sign In

This topic created in 4194 days ago, the information mentioned may be changed or developed.

Error: java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.io.Text, received org.apache.hadoop.io.LongWritable
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1069)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:703)
at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:775)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

这里是Error，因为初学MapReduce所以不很弄的清楚他的Type

mapreduce

type

mapper

6 replies • 2014-11-17 12:51:59 +08:00

libo26

Nov 16, 2014

这不很明显么。。
expected org.apache.hadoop.io.Text, received org.apache.hadoop.io.LongWritable
把你的mapper函数贴出来看看？

SharkIng

Nov 16, 2014

https://gist.github.com/SharkIng/2b86024180b2a5e93e6a

SharkIng

Nov 16, 2014

@libo26 求教了，非常感谢。
写的头都大了还是不知道问题出在哪里了，初学者多多指教

lcj2class

Nov 16, 2014

从map函数的定义<LongWritable, Text, Text, IntWritable>来看是没有错的,
你的main函数里面是怎么设置map的输出类型的呢?

SharkIng

Nov 17, 2014

@lcj2class

job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);

// TODO: Need add args for input.
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);

感觉所有地方都没错所以不知道是怎么会有这么一个Error

lcj2class

Nov 17, 2014

@SharkIng
把整个代码都贴出来吧