The reports should provide the following information:

• A written description of data with relevant spreadsheets.
• Explanation of how you analysed your data.
• Explanation of what data you analysed and follow with relevant visualization.
• Show the results of your analysis, highlight important results.
• Details for each task required in the project.

College of Computing and Informatics

2020/2021 Second Semester

Course Code

DS520

Course Name

Big Data Processing and Analytics

CRN

Assignment type

Critical Thinking Project

Module

All modules

Assignment Points

10

Project Template

Task 1:

1.1 Literature Review:

———————————————————–

1.2 References:

———————————————————–

Task 2:

2.1 Introduction

Provide a short description of your project and an overview about the data you are analysing.

2.2 Body section

2.2.1 Data

This section should include a description of the data being analyse (include number of samples in the dataset, features and their types, descriptive statistics of the data, etc).

2.2.2 Steps:

In this section, write the steps and commands you used to import the data and.

Task 3:

3.1 MapReduce Algorithm (Comment your Code)

Write the complete code you applied.

3.2 Results

Include a written description of the statistical results, and its meaning based on the dataset you have chosen.

Task 4:

4.1 Steps:

In this section, write the steps and commands you used to import the data and.

Task 5:

5.1 Applied Queries on MongoDB

Write the complete code you applied with describing the function of each query.

5.2 Results

Include a written description of the results. Discuss the meaning of the results based on the data set.

Task 6:

6.1 Applied Code on Hive/Pig

Write the complete code you applied with describing the function of each query.

6.2 Results

Include a written description of the results. Discuss the meaning of the results based on the data set.

Task 7:

6.1 Applied Code on SparkSQL

Write the complete code you applied with describing the function of each query.
6.2 Results

Include a written description of the results. Discuss the meaning of the results based on the data set (Include visualization of the results) Figures must be added.

Task 8:

8.1 Applied Code on Spark (Using MLib)

Write the complete code you applied with describing the machine learning algorithm and why you choose it.

8.2 Results

Include a written description of the results. Discuss the meaning of the results based on the data set.

Conclusion

Restate the main results of your analysis and provide any future recommendations.

College of Computing and Informatics

Project Dataset:

1-

https://www.kaggle.com/austinreese/craigslist-carstrucks-data

2-

https://www.kaggle.com/currie32/crimes-in-chicago

3-

https://www.kaggle.com/hm-land-registry/uk-housing-prices-paid

You can choose any one of the previous datasets. And apply all the following tasks on the dataset you choose.

Project Required Steps:

Task 1: (2 Marks)

Topic 1: Sentiment analysis is used in identifying the public opinion through text analytics. Big data tools can aid in the storage and processing of data for sentiment analysis. Through such analysis, companies can better plan their processes and sales accordingly.

Topic 2: Machine Learning algorithms are very important in the field of data science. With the increasing number of data, it is very important and advantageous to apply those algorithms on Big Data.

Write a small Literature Review and discussion about topic 1 or topic 2 discussing how this topic can be implemented and used in Big Data applications, in no more than one paper. You must use at least six references and cite them in the Literature Review. The reference must be added to the template (Try using any referencing software).

Task 2: (1 Marks)

Load the data set into Hadoop File System. Discuss and explain the type and structure of the data. Show the steps that you followed during the importing process.

Task 3: (2 Marks)

Apply Map Reduce algorithm to produce useful statistical results. Discuss in detail the statistical results, and its meaning based on the dataset you have chosen.

Task 4: (1 Marks)

Import the data in MongoDB. Show the steps you followed to import the dataset to any of these NoSQL systems.

Task 5: (2 Marks)

Execute at least three queries on the data MongoDB. Describe your queries and the results. Discuss the meaning of the results based on the data set.

Task 6: (1 Marks)

Using Hive or Pig, execute at least three queries on the data set. Describe your queries and the results. Discuss the meaning of the results based on the data.

Task 7: (1 Marks)

Using Spark, run two SparkSQL statements on the dataset, and visualize the results in any of the charts (Hints: you can use Zeppelin directly).

Task 8 (Optional): (1 Marks as Bonus)

Using Mlib in Spark, build a suitable machine learning model and execute it on the data. Discuss your results.

Note:

· You can use Horton HDP sandbox with only one node. For the part on Spark you can use the same sandbox, or you can use Databricks cluster.

· All the tasks must be described in detail with the code written for each part.

· You can add screenshots of your steps to the project template.