Getting error in Custom Input Format

0 votes

I have a file in which I have ‘^’ character. I have written a custom input format where I have used ‘^’ character as delimiter. In my custom input format, I am extending FileInputFormat and I have also used custom record reader where I am extending RecordReader. I an getting an error in the nextKeyValue() method in the while loop. Can anyone help me in how to read data, split it and generate key value pair?

public class CustomRecordReader extends RecordReader<LongWritable, Text>

    {
        long start, current, end;
        Text value;
        LongWritable key;
        LineReader reader;
        FileSplit split;
        Path path;
        FileSystem fs;
        FSDataInputStream in;
        Configuration conf;

        @Override
        public void initialize(InputSplit inputSplit, TaskAttemptContext cont) throws IOException, InterruptedException

        {
            conf = cont.getConfiguration();
            split = (FileSplit)inputSplit;
            path = split.getPath();
            fs = path.getFileSystem(conf);
            in = fs.open(path);
            reader = new LineReader(in, conf);
            start = split.getStart();
            current = start;
            end = split.getLength() + start;
        }

        @Override
        public boolean nextKeyValue() throws IOException
        {
            if(key==null)
                key = new LongWritable();
            key.set(current);
            if(value==null)
                value = new Text();
            long readSize = 0;

            while(current<end)
            {
                Text tmpText = new Text();
                readSize = read //here how should i read data from the split, and generate key-value?
                if(readSize==0)
                    break;
                current+=readSize;             
            }

            if(readSize==0)
            {
                key = null;
                value = null;
                return false;
            }
            return true;

        }

        @Override
        public float getProgress() throws IOException
        {
        }

        @Override
        public LongWritable getCurrentKey() throws IOException
        {
        }

        @Override
        public Text getCurrentValue() throws IOException
        {
        }

        @Override
        public void close() throws IOException
        {
        }
    }
Apr 19, 2018 in Big Data Hadoop by coldcode
• 2,010 points
27 views

1 answer to this question.

0 votes

Use this property directly to delimit the records:

conf.set("textinputformat.record.delimiter", "^");
answered Apr 19, 2018 by Shubham
• 13,190 points

Related Questions In Big Data Hadoop

0 votes
1 answer

Getting error in MapReduce job.setInputFormatClass

In old Hadoop API(i.e. below Hadoop 0.20.0), ...READ MORE

answered Apr 15, 2018 in Big Data Hadoop by Shubham
• 13,190 points
268 views
0 votes
1 answer

Getting error in Hadoop Streaming: Type mismatch in Key from Map

In Hadoop streaming you have to customize ...READ MORE

answered Apr 18, 2018 in Big Data Hadoop by coldcode
• 2,010 points
203 views
0 votes
1 answer

Getting error in Hadoop: Output file already exist

When you executed your code earlier, you ...READ MORE

answered Apr 19, 2018 in Big Data Hadoop by Shubham
• 13,190 points
1,107 views
0 votes
1 answer

Getting error in JAVA_HOME variable.

First of all check the hadoop version ...READ MORE

answered May 11, 2018 in Big Data Hadoop by Shubham
• 13,190 points
104 views
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,110 points
2,054 views
0 votes
10 answers

hadoop fs -put command?

copy command can be used to copy files ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Sujay
10,506 views
0 votes
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,240 points
765 views
0 votes
1 answer
0 votes
1 answer
0 votes
1 answer

Getting error in command hdfs dfs -mkdir

You are getting this error because the ...READ MORE

answered Apr 15, 2018 in Big Data Hadoop by Shubham
• 13,190 points
383 views