ioexception - Hadoop can't finish job because "No space left on device" -

i trying run simple hadoop job. modification of classic wordcount which, instead of counting words, counts lines in file. want use clean bunch of big log files (around 70gb each) know have duplications. each line "record", , hence interested in getting each record once.

i know code works, because should when run small normal files. when run big files, hadoop behaves stringently. first, starts correctly working on map phase, reaches 100% without problems. when dealing reduce, however, never reaches more 50%. reaches maybe 40%, , go 0% after showing "no space left on device" exceptions:

fserror: java.io.ioexception: no space left on device 

then tries reduce again and, when reaches 40%, drops 0% again , forth. 2 or 3 times before decides end without success, of course.

the problem exception, though, can't related actual space on disks. disk space never gets full. not total (global) space on hdfs, neither individual disks in each node. check fs status with:

$ hadoop dfsadmin -report > report 

this report never shows actual node reaching 100%. in fact, no node come close that.

i have around 60gb of disk available in each node me, , run in cluster 60 data nodes, gives me total space of more 3tb. file trying process 70gb.

looking out there on internet, found can related hadoop creating many files while processing lot of data. original wordcount code reduces data substantially (since words repeat lot). file 70gb can reduced output of 7mb. however, expecting 1/3 reduction only, or output of around 20-30gb.

unix-type systems come limit of 1024 open files per process:

$ ulimit -n 1024 

if hadoop creating more that, problem. asked system admin increment limit 65k, limit now:

$ ulimit -n 65000 

problems continue. can need increment limit further? there else going on here?

thanks lot help!

code here:

package ...;  import java.io.ioexception; import java.util.stringtokenizer;  import org.apache.hadoop.conf.configuration; import org.apache.hadoop.fs.path; import org.apache.hadoop.io.intwritable; import org.apache.hadoop.io.text; import org.apache.hadoop.mapreduce.job; import org.apache.hadoop.mapreduce.mapper; import org.apache.hadoop.mapreduce.reducer; import org.apache.hadoop.mapreduce.lib.input.fileinputformat; import org.apache.hadoop.mapreduce.lib.output.fileoutputformat; import org.apache.hadoop.util.genericoptionsparser;  public class linecountmr {    public static class mapperclass         extends mapper<object, text, text, intwritable>{      private final static intwritable 1 = new intwritable(1);     private text word = new text();     private string token = new string();              public void map(object key, text value, context context                     ) throws ioexception, interruptedexception {          token = value.tostring().replace(' ', '_');         word.set(token);         context.write(word, one);        }   }    public static class reducerclass         extends reducer<text,intwritable,text,intwritable> {     private intwritable result = new intwritable();      public void reduce(text key, iterable<intwritable> values,                         context context                        ) throws ioexception, interruptedexception {       int sum = 0;       (intwritable val : values) {         sum += val.get();       }       result.set(sum);       context.write(key, result);     }  }    public static void main(string[] args) throws exception {     configuration conf = new configuration();;     if (args.length != 2) {       system.err.println("parameters: <in> <out>");       system.exit(2);     }     job job = new job(conf, "line count mr");     job.setjarbyclass(linecountmr.class);     job.setmapperclass(mapperclass.class);     job.setcombinerclass(reducerclass.class);     job.setreducerclass(reducerclass.class);     job.setoutputkeyclass(text.class);     job.setoutputvalueclass(intwritable.class);     fileinputformat.addinputpath(job, new path(args[0]));     fileoutputformat.setoutputpath(job, new path(args[1]));     system.exit(job.waitforcompletion(true) ? 0 : 1);   } } 

i have seen issue on cluster while processing 10tb of data. issue not related space availability on hdfs space available on local file system (df -h) used storing intermediate data generated during map-reduce operation stored locally , not in hdfs.