Number of tasks submitted and executed within the day in hadoop

0 votes
Tasks

Write a software application, service, daemon or web application which will accept requests from remote users in order to trigger the execution of shell scripts which have been already deployed in the machine and the module is aware of their location and also naming.  

The scripts should accept parameters which are passed by the module. We recommend to model this behavior with a shell script which will accept strings as arguments and write them in a text file.  

The remote user can transfer files to a local directory and pickup result files if any. No SSH shell access should be invoked in the process. It can be considered that the module is operating in a highly trusted environment so no authentication is required in the context of the exercise.  

Any file output of the shell script's execution should be stored retrieved from the query and stored in a directory defined by you. You can assume that there is full transparency in the location of the script's output, you can make it the easiest way possible for you.

Any output in the standard output should be stored in a separate file, in order to identify later any possible error in the script's execution.

The bonus part of task will be to add the collection of KPIs and visualize them with any tool of your choice. The KPIs we recommend you to collect are the following:

• Timestamp or date time of the execution for every task submitted.

• Number of tasks submitted and executed within the day.

• Free disk space after the execution of every submitted task (remember that in production this module is handling files with 100s of Gigabytes of size so this metric is crucial).

Visualizing the KPIs with any tool of your choice will be the icing on the cake. :-)

Conditions

• You can use any language and framework you wish and host you wish.

• You should not publish your solution online or by any other means without Ooredoo's consent.

• Style and sane principles will be appreciated.

• Submitting your solution after the deadline will render it invalid.
Dec 12, 2018 in Big Data Hadoop by Irfan
• 120 points
843 views
Hello @Irfan. Can you tell me what you have tried so far? Are you stuck at a particular implementation?

No answer to this question. Be the first to respond.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.

Related Questions In Big Data Hadoop

0 votes
1 answer

Method to schedule the number of Mappers and Reducers in a Hadoop MapReduce Tsk.

The number of Mapper tasks for a ...READ MORE

answered Apr 25, 2019 in Big Data Hadoop by ravikiran
• 4,620 points
806 views
0 votes
1 answer

Setting the Number of Map tasks and Reduce Tasks not Working

The command that you are running is ...READ MORE

answered May 4, 2018 in Big Data Hadoop by Ashish
• 2,650 points
1,496 views
0 votes
1 answer
0 votes
1 answer

I have to ingest in hadoop cluster large number of files for testing , what is the best way to do it?

Hi@sonali, It depends on what kind of testing ...READ MORE

answered Jul 8, 2020 in Big Data Hadoop by MD
• 95,460 points
1,240 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
11,065 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
2,566 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
109,009 views
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,350 points
4,632 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP