Getting error in Custom Input Format

0 votes

I have a file in which I have ‘^’ character. I have written a custom input format where I have used ‘^’ character as delimiter. In my custom input format, I am extending FileInputFormat and I have also used custom record reader where I am extending RecordReader. I an getting an error in the nextKeyValue() method in the while loop. Can anyone help me in how to read data, split it and generate key value pair?

public class CustomRecordReader extends RecordReader<LongWritable, Text>

    {
        long start, current, end;
        Text value;
        LongWritable key;
        LineReader reader;
        FileSplit split;
        Path path;
        FileSystem fs;
        FSDataInputStream in;
        Configuration conf;

        @Override
        public void initialize(InputSplit inputSplit, TaskAttemptContext cont) throws IOException, InterruptedException

        {
            conf = cont.getConfiguration();
            split = (FileSplit)inputSplit;
            path = split.getPath();
            fs = path.getFileSystem(conf);
            in = fs.open(path);
            reader = new LineReader(in, conf);
            start = split.getStart();
            current = start;
            end = split.getLength() + start;
        }

        @Override
        public boolean nextKeyValue() throws IOException
        {
            if(key==null)
                key = new LongWritable();
            key.set(current);
            if(value==null)
                value = new Text();
            long readSize = 0;

            while(current<end)
            {
                Text tmpText = new Text();
                readSize = read //here how should i read data from the split, and generate key-value?
                if(readSize==0)
                    break;
                current+=readSize;             
            }

            if(readSize==0)
            {
                key = null;
                value = null;
                return false;
            }
            return true;

        }

        @Override
        public float getProgress() throws IOException
        {
        }

        @Override
        public LongWritable getCurrentKey() throws IOException
        {
        }

        @Override
        public Text getCurrentValue() throws IOException
        {
        }

        @Override
        public void close() throws IOException
        {
        }
    }
Apr 19, 2018 in Big Data Hadoop by coldcode
• 2,080 points
201 views

1 answer to this question.

0 votes

Use this property directly to delimit the records:

conf.set("textinputformat.record.delimiter", "^");
answered Apr 19, 2018 by Shubham
• 13,490 points

Related Questions In Big Data Hadoop

0 votes
1 answer

Getting error in MapReduce job.setInputFormatClass

In old Hadoop API(i.e. below Hadoop 0.20.0), ...READ MORE

answered Apr 15, 2018 in Big Data Hadoop by Shubham
• 13,490 points
1,201 views
0 votes
1 answer

Getting error in Hadoop Streaming: Type mismatch in Key from Map

In Hadoop streaming you have to customize ...READ MORE

answered Apr 18, 2018 in Big Data Hadoop by coldcode
• 2,080 points
654 views
0 votes
1 answer

Getting error in Hadoop: Output file already exist

When you executed your code earlier, you ...READ MORE

answered Apr 19, 2018 in Big Data Hadoop by Shubham
• 13,490 points
5,934 views
0 votes
1 answer

Getting error in JAVA_HOME variable.

First of all check the hadoop version ...READ MORE

answered May 11, 2018 in Big Data Hadoop by Shubham
• 13,490 points
1,207 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
8,762 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
78,115 views
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
3,349 views
0 votes
1 answer
0 votes
1 answer
0 votes
1 answer

Getting error in command hdfs dfs -mkdir

You are getting this error because the ...READ MORE

answered Apr 15, 2018 in Big Data Hadoop by Shubham
• 13,490 points
1,578 views
webinar REGISTER FOR FREE WEBINAR X
Send OTP
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP