How to count lines in a file on hdfs command?

0 votes

I have a file on HDFS that I want to know how many lines are. (testfile)

In linux, I can do:

wc -l <filename>

Can I do something similar with "hadoop fs" command? I can print file contents with:

hadoop fs -text /user/mklein/testfile

How do I know how many lines do I have? I want to avoid copying the file to local filesystem then running the wc command.

Note: My file is compressed using snappy compression, which is why I have to use -text instead of -cat

Aug 9, 2018 in Big Data Hadoop by Frankie
• 9,710 points
1,635 views

1 answer to this question.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
0 votes

Use the below commands:

Total number of files: hadoop fs -ls /path/to/hdfs/* | wc -l

Total number of lines: hadoop fs -cat /path/to/hdfs/* | wc -l

Total number of lines for a given file: hadoop fs -cat /path/to/hdfs/filename | wc -l

answered Aug 10, 2018 by Neha
• 6,180 points

Related Questions In Big Data Hadoop

0 votes
1 answer

What is the command to count number of lines in a file in hdfs?

hadoop fs -cat /example2/doc1 | wc -l READ MORE

answered Nov 22, 2018 in Big Data Hadoop by Omkar
• 67,000 points
164 views
0 votes
1 answer

How to print the content of a file in console present in HDFS?

Yes, you can use hdfs dfs command ...READ MORE

answered Apr 19, 2018 in Big Data Hadoop by Shubham
• 12,890 points
309 views
0 votes
1 answer

How to write a file in hdfs with Java?

You could pass the URI when getting ...READ MORE

answered Sep 26, 2018 in Big Data Hadoop by digger
• 27,620 points
52 views
0 votes
1 answer

How to write a file in HDFS using Java Programming language?

Define the HADOOP_CONF_DIR environment variable to your Hadoop configuration ...READ MORE

answered Sep 28, 2018 in Big Data Hadoop by Frankie
• 9,710 points
85 views
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 9,670 points
1,879 views
0 votes
10 answers

hadoop fs -put command?

copy command can be used to copy files ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Sujay
9,391 views
0 votes
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,240 points
676 views
0 votes
1 answer
0 votes
1 answer

How to create a project for the first time in Hadoop.?

If you want to learn Hadoop framework ...READ MORE

answered Jul 26, 2018 in Big Data Hadoop by Neha
• 6,180 points
43 views
0 votes
1 answer

How to convert a string to timestamp with milliseconds in Hive?

 concatenation of substrings using the following code: select ...READ MORE

answered Oct 31, 2018 in Big Data Hadoop by Neha
• 6,180 points
998 views

© 2018 Brain4ce Education Solutions Pvt. Ltd. All rights Reserved.
"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.