29538/spark-scala-how-to-list-all-folders-in-directory
I can do this by using the command in Hadoop:
hadoop fs -ls hdfs://sandbox.hortonworks.com/demo/
I want to do the same with Scale or Spark. Can someone tell me how to do it?
This should work
val fs = FileSystem.get(new Configuration()) val status = fs.listStatus(new Path(YOUR_HDFS_PATH)) status.foreach(x=> println(x.getPath))
If you want to know more about Apache Spark Scala, It's highly recommended to go for Spark certification course today.
Thanks!!
val listStatus = org.apache.hadoop.fs.FileSystem.get(new URI(url), sc.hadoopConfiguration) .globStatus(new org.apache.hadoop.fs.Path(url)) for (urlStatus <- listStatus) { println("urlStatus get Path:" + urlStatus.getPath()) }
val spark = SparkSession.builder().appName("Demo").getOrCreate() val path = new Path("enter your directory path") val fs:FileSystem = projects.getFileSystem(spark.sparkContext.hadoopConfiguration) val it = fs.listLocatedStatus(path)
Well, what you can do is use ...READ MORE
You can try filter using value in ...READ MORE
I think the exception is caused because ...READ MORE
Just try the following command: hadoop fs -rm ...READ MORE
Instead of spliting on '\n'. You should ...READ MORE
Firstly you need to understand the concept ...READ MORE
org.apache.hadoop.mapred is the Old API org.apache.hadoop.mapreduce is the ...READ MORE
Hi, You can create one directory in HDFS ...READ MORE
There is no particular format to handle ...READ MORE
Hi, You can try this command: hadoop fs -ls ...READ MORE
OR
At least 1 upper-case and 1 lower-case letter
Minimum 8 characters and Maximum 50 characters
Already have an account? Sign in.