Getting error in Custom Input Format

0 votes

I have a file in which I have ‘^’ character. I have written a custom input format where I have used ‘^’ character as delimiter. In my custom input format, I am extending FileInputFormat and I have also used custom record reader where I am extending RecordReader. I an getting an error in the nextKeyValue() method in the while loop. Can anyone help me in how to read data, split it and generate key value pair?

public class CustomRecordReader extends RecordReader<LongWritable, Text>

    {
        long start, current, end;
        Text value;
        LongWritable key;
        LineReader reader;
        FileSplit split;
        Path path;
        FileSystem fs;
        FSDataInputStream in;
        Configuration conf;

        @Override
        public void initialize(InputSplit inputSplit, TaskAttemptContext cont) throws IOException, InterruptedException

        {
            conf = cont.getConfiguration();
            split = (FileSplit)inputSplit;
            path = split.getPath();
            fs = path.getFileSystem(conf);
            in = fs.open(path);
            reader = new LineReader(in, conf);
            start = split.getStart();
            current = start;
            end = split.getLength() + start;
        }

        @Override
        public boolean nextKeyValue() throws IOException
        {
            if(key==null)
                key = new LongWritable();
            key.set(current);
            if(value==null)
                value = new Text();
            long readSize = 0;

            while(current<end)
            {
                Text tmpText = new Text();
                readSize = read //here how should i read data from the split, and generate key-value?
                if(readSize==0)
                    break;
                current+=readSize;             
            }

            if(readSize==0)
            {
                key = null;
                value = null;
                return false;
            }
            return true;

        }

        @Override
        public float getProgress() throws IOException
        {
        }

        @Override
        public LongWritable getCurrentKey() throws IOException
        {
        }

        @Override
        public Text getCurrentValue() throws IOException
        {
        }

        @Override
        public void close() throws IOException
        {
        }
    }
Apr 19, 2018 in Big Data Hadoop by coldcode
• 2,080 points
456 views

1 answer to this question.

0 votes

Use this property directly to delimit the records:

conf.set("textinputformat.record.delimiter", "^");
answered Apr 19, 2018 by Shubham
• 13,490 points

Related Questions In Big Data Hadoop

0 votes
1 answer

Getting error in MapReduce job.setInputFormatClass

In old Hadoop API(i.e. below Hadoop 0.20.0), ...READ MORE

answered Apr 15, 2018 in Big Data Hadoop by Shubham
• 13,490 points
1,817 views
0 votes
1 answer

Getting error in Hadoop Streaming: Type mismatch in Key from Map

In Hadoop streaming you have to customize ...READ MORE

answered Apr 18, 2018 in Big Data Hadoop by coldcode
• 2,080 points
999 views
0 votes
1 answer

Getting error in Hadoop: Output file already exist

When you executed your code earlier, you ...READ MORE

answered Apr 19, 2018 in Big Data Hadoop by Shubham
• 13,490 points
7,993 views
0 votes
1 answer

Getting error in JAVA_HOME variable.

First of all check the hadoop version ...READ MORE

answered May 11, 2018 in Big Data Hadoop by Shubham
• 13,490 points
1,955 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
10,521 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
103,814 views
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
4,234 views
0 votes
1 answer
0 votes
1 answer
0 votes
1 answer

Getting error in command hdfs dfs -mkdir

You are getting this error because the ...READ MORE

answered Apr 15, 2018 in Big Data Hadoop by Shubham
• 13,490 points
2,780 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP