What are the various ways to import files into HDFS

0 votes
I am new to Hadoop and I want to know what are the different ways through which I can bring data into HDFS.

I have a web application that is creating log files, how can i import the log files into HDFS?
Apr 13, 2018 in Big Data Hadoop by Shubham
• 13,490 points
3,605 views

1 answer to this question.

0 votes

There are various tools and frameworks available in Hadoop Ecosystem, using which you can import and export data to HDFS & from HDFS respectively.

let me tell you few of them and their use cases:

  1. hdfs dfs -put - simple way to insert files from local file system to HDFS
  2. HDFS Java API
  3. Sqoop - for bringing data to/from databases
  4. Flume - streaming files, logs
  5. Kafka - distributed queue, mostly for near-real time stream processing
  6. Nifi - incubating project at Apache for moving data into HDFS without making lots of changes

So for your purpose, the best solution for bringing web application logs to HDFS is through Flume.

answered Apr 13, 2018 by nitinrawat895
• 11,380 points

Related Questions In Big Data Hadoop

+1 vote
3 answers

What is the best way to merge multi-part HDFS files into single file?

1. In order to merge two or ...READ MORE

answered Jul 29, 2019 in Big Data Hadoop by Tina
31,041 views
0 votes
5 answers
0 votes
1 answer

What are the different ways to load data from Hadoop to Azure Data Lake?

I would recommend you to go through ...READ MORE

answered Apr 18, 2018 in Big Data Hadoop by coldcode
• 2,090 points
1,066 views
0 votes
1 answer

What are the different ways of Installing Hadoop into our local machine?

Hadoop runs on Unix and on Windows. ...READ MORE

answered Aug 4, 2018 in Big Data Hadoop by Neha
• 6,300 points
5,281 views
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,350 points
4,635 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
11,070 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
109,024 views
0 votes
1 answer
0 votes
1 answer

Copy file from HDFS to the local file system

There are two possible ways to copy ...READ MORE

answered Mar 27, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
17,165 views
0 votes
11 answers

What is the default Namenode port of HDFS. Is it 8020 or 9000 or 50070?

8020/9000 hadoop namenode metadata port number. or ...READ MORE

answered Dec 31, 2018 in Big Data Hadoop by SM
63,092 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP