pfa

Topic: one page excluding reference for every posted article as below requirement

From the posted journal topic create the new topic(research title) needs to have a gap in the literature that leads to the research question.

The topic proposal is a general sketch of the dissertation – the topic, general reasoning behind the topic, as well as a potential thesis and thesis map for your Literature Review.  The topic proposal must clearly link to theory, identify what problem or gap in literature your proposed topic will address, and show a connection to program goals and core courses.

Requirement:

APA format

Grammarly check

No plagiarism

If above topic complete will think below on the same for further study

Here are some things to consider for the future research:

Identify a topic in your specialization

What is the cited problem? The problem you want to address has to come from the literature, and be cited, who are saying there are problems.

What is the population you would like to address?

What methodology are you planning on using? (If you are unsure, do you plan on a quantitative or qualitative study?  You will determine your methodology later).

Locate at least one article related to your topic. What was the research question in the study? Who were the participants in the study? What findings were reported?

Received January 19, 2021, accepted February 1, 2021, date of publication February 22, 2021, date of current version March 2, 2021.

Digital Object Identifier 10.1109/ACCESS.2021.3060863

The Role of AI, Machine Learning, and Big Data in
Digital Twinning: A Systematic Literature Review,
Challenges, and Opportunities
M. MAZHAR RATHORE 1, (Member, IEEE), SYED ATTIQUE SHAH2, DHIRENDRA SHUKLA3,
ELMAHDI BENTAFAT1, AND SPIRIDON BAKIRAS 1, (Member, IEEE)
1Division of Information and Computing Technology, College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
2Data Systems Group, Institute of Computer Science, University of Tartu, 51009 Tartu, Estonia
3Dr. J. Herbert Smith Centre, University of New Brunswick, Fredericton, NB E3B 5A3, Canada

Corresponding author: Spiridon Bakiras (sbakiras@hbku.edu.qa)

ABSTRACT Digital twinning is one of the top ten technology trends in the last couple of years, due to its high
applicability in the industrial sector. The integration of big data analytics and artificial intelligence/machine
learning (AI-ML) techniques with digital twinning, further enriches its significance and research potential
with new opportunities and unique challenges. To date, a number of scientific models have been designed
and implemented related to this evolving topic. However, there is no systematic review of digital twinning,
particularly focusing on the role of AI-ML and big data, to guide the academia and industry towards future
developments. Therefore, this article emphasizes the role of big data and AI-ML in the creation of digital
twins (DTs) or DT-based systems for various industrial applications, by highlighting the current state-of-
the-art deployments. We performed a systematic review on top of multidisciplinary electronic bibliographic
databases, in addition to existing patents in the field. Also, we identified development-tools that can facilitate
various levels of the digital twinning. Further, we designed a big data driven and AI-enriched reference
architecture that leads developers to a complete DT-enabled system. Finally, we highlighted the research
potential of AI-ML for digital twinning by unveiling challenges and current opportunities.

INDEX TERMS Digital twin, artificial intelligence, machine learning, big data, industry 4.0.

I. INTRODUCTION
Digital twinning is a process that involves the creation of a
virtual model (i.e., a twin) of any physical object, in order
to streamline, optimize, and maintain the underlying physi-
cal process. Theoretically, the digital twin concept was first
presented in 2002 by Grieves et al. [1] during a special
meeting on product life-cycle management at the University
of Michigan Lurie Engineering Center. In his subsequent
article [2], he further defined digital twinning as a combi-
nation of three primary components: 1) a virtual twin; 2) a
corresponding physical twin (a physical object that can be
a product, a system, a model, or any other component such
as, a robot, a car, a power turbine, a human, a hospital, etc.);
and 3) a data flow cycle that feeds data from a physical twin
to its virtual twin and takes back the information and pro-
cesses from the virtual twin to the physical twin. The virtual

The associate editor coordinating the review of this manuscript and

approving it for publication was Claudio Zunino .

twin is nothing but an algorithm that replicates the behavior
(fully or partially) of the corresponding physical counterpart,
by generating the same output as does the physical object on
given input values. Mostly, it is considered as part of the smart
manufacturing process, but it can be used in any domain, such
as construction, education, business, transport, power and
electronics, human and healthcare, sports, and networking
and communications.

Digital twinning was first adopted by Tuegel et al. [3]
in 2011 to digitally reproduce the structural behavior of an
aircraft. Initially, digital twinning was used as a mainte-
nance tool to continuously monitor the craft’s structure. Then,
it was replicated as a complete twin in order to simulate
its entire life-cycle and predict its performance [3]. Later,
digital twinning started gaining popularity in several indus-
tries that aimed at making their processes smarter, intelligent,
and optimally dynamic, based on the operating conditions.
The technology raises its global demand, as it facilitates in
finding the product flaws, reducing production cost, real-time

32030 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ VOLUME 9, 2021

https://orcid.org/0000-0002-0701-9850

https://orcid.org/0000-0002-8964-0746

https://orcid.org/0000-0001-7619-5412

M. M. Rathore et al.: Role of AI, Machine Learning, and Big Data in Digital Twinning: A SLR, Challenges, and Opportunities

monitoring of resources, and increasing the life of the product
by predicting product failure. On this account, digital twin-
ning became one of the top-ten technology trends [4].

Several surveys have been published, highlighting the cur-
rent research trends of digital twinning in various fields.
For instance, Wanasinghe et al. [5] pointed out the state-of-
the-art works of digital twinning in the oil and gas indus-
tries. Lu et al. [6] and Cimino et al. [7] reviewed the
current reference models, applications, and research issues
in manufacturing. Qi and Tao [8] emphasized on the role of
data and digital twinning in achieving smart manufacturing.
DT-related patents are discussed by Tao et al. [9] in different
industries. And, the modeling perspective of digital twinning
is explored by Rasheed et al. [10].
Recently, the use of IoT, big data, and AI-ML technologies

have brought new potentials in digital twinning. The adoption
of these techniques ensures a perfect digital twin and intro-
duces new research challenges and opportunities. Since 2015,
several digital twins have been developed in various indus-
tries using AI-ML and big data analytics, and the number of
related research articles is growing rapidly. Despite the grow-
ing popularity, adaptability, and applicability of AI-enabled
digital twinning in the industrial sector, exploited by IoT
and big data technologies, no systematic review has been
performed that explicitly focuses on the role of these tech-
nologies in digital twinning. The above-mentioned surveys
do not fully cover the importance of these technologies in the
DT domain. Therefore, there is an exigency of a systematic
approach towards the thorough review of the current develop-
ments in AI-enabled digital twinning using IoT technology
and big data. This can drive both academia and industry
towards further research, by highlighting the current findings,
future potentials, challenges, and applications of AI-enabled
digital twinning in the industrial sector.

In this article, we carried out a systematic literature review
that incorporates all the research work in the form of articles,
patents, and web-reports, covering digital twinning and its
integration with state-of-the-art AI-ML and big data analytics
techniques. We highlighted the role of big data, AI, machine
learning, and IoT technologies in the process of digital twin
creation, by listing examples from current deployments in
various industrial domains. We introduced the digital twin
paradigm, by explaining its basic concepts and highlighting
its applications in several industrial areas. After a thorough
literature survey, we identified 1) tools that can be used for
digital twin creation; 2) the criteria for successful digital twin-
ning; and 3) research opportunities and challenges in digital
twinning for diverse industrial sectors. Finally, we designed
a reference model for digital twinning that exploits IoT, big
data, and AI-ML approaches.

The rest of the paper is organized as follows. Section II
briefly presents the survey methodology. Section III formally
defines digital twinning, its creation method, and other basic
concepts. Section IV summarizes the application of digital
twinning in various industries. Section V briefly describes
big data and AI, while Section VI discusses the relationship

between IoT, big data, AI, and digital twinning. Section VII
summarizes the role of AI in digital twinning with state-
of-the-art research developments. Section VIII outlines the
important data-driven patents in digital twinning. Section IX
presents the evaluation criteria for an ideal digital twin-
ning, and Section X lists the tools that may be required in
the process of digital twinning. The design details of the
reference architecture for AI-enabled DT creation is pre-
sented in Section XI, while the current research opportunities
and research challenges in digital twinning are described in
Section XII. The article is concluded in Section XIII.

II. METHODOLOGY
To the best of our knowledge, the survey at hand is the
first of its kind in terms of reviewing AI-ML and big data
analytics techniques for digital twinning. The systematic lit-
erature review (SLR) carried out in this study is based on
the guidelines recommended by [11], [12], with the aim of
summarizing the current literature and establishing the basis
for qualitative synthesis and information extraction. SLR is
an organized, efficient, and widely recognized method that
is comparatively better than the traditional literature review
process [13].

We identified the following six research questions that
directed our entire review process:

1) What is digital twinning, how does it work, and what
are the standards and technologies to create a digital
twin (DT)?

2) What is the relationship between AI-ML, big data, IoT,
and digital twinning?

3) What is the role of AI-ML and big data analytics in
digital twinning, its related applications, and current
deployments in different industrial sectors?

4) What are the tools required for the creation of
AI-enabled DT?

5) What is the criteria for a successful DT or DT-based
system?

6) What are the main challenges, market opportunities,
and future directions in digital twinning?

To capture the wide range of digital twinning applica-
tions, we searched eight multidisciplinary electronic biblio-
graphic databases, including 1) IEEE Xplore (IEEE, IET);
2) ACM digital library; 3) Scopus (ScienceDirect, Else-
vier); 4) SpringerLink (Springer); 5) Hindawi; 6) IGI-Global;
7) Taylor & Francis online; and 8) Wiley online library.
We also searched the US patents database. Using suitable
search strings is crucial to extracting the appropriate liter-
ature from the electronic bibliographic databases. Due to
the diverse nature of this study, we used a set of appro-
priate keywords that assures the inclusion of AI-ML and
big data analytics in industrial digital twinning. Specifically,
as shown in Table 1, we defined various keywords, combined
with logical operators, to search the electronic bibliographic
databases.

The search was carried out just before August 2020. Prior
to 2015, we found very few papers on digital twinning.

VOLUME 9, 2021 32031

M. M. Rathore et al.: Role of AI, Machine Learning, and Big Data in Digital Twinning: A SLR, Challenges, and Opportunities

TABLE 1. Search strings.

FIGURE 1. Number of journal papers published by different libraries.

In 2017, the topic gained popularity and became one of the
top 10 trends in strategic technology [14]–[16]. In the period
2015–2020, more than 2000 Scopus-indexed journal articles,
more than 1000 patents, 250 book chapters, and 20 books
have been published, discussing digital twinning technology.
However, we identified over 850 articles that match the search
criteria defined in Table 1. Fig. 1 and 2 show the total
number of journal and conference papers published on the
topic of digital twinning by the different libraries. Among
other publishers, IGI-Global published seven articles, Hin-
dawi published three articles, and ACM published only two
articles in their journals. Additionally, Fig. 3 illustrates the pie
chart of published articles related to various applications of
DT (it includes both conference and journal papers). Clearly,
manufacturing is the dominant application area for digital
twinning.

Considering the aforementioned research questions,
we defined a set of inclusion and exclusion criteria for an
article as follows:

1) The study is written in English.
2) The study is published in a scientific journal, magazine,

book, book chapter, conference, or workshop.
3) The journal article is included only if the journal’s

impact factor is > 1.0.
4) The conference article is included only if the confer-

ence is mature enough (it has already published at least
fifteen versions of its proceedings).

5) Publications such as dissertations, in-progress research,
guest editorials, poster sessions, and blogs are
excluded.

6) Duplicate papers that appear in several electronic
databases will only be considered once.

7) The study is excluded if not fully focusing on the digital
twinning concept or any of its specified applications.

FIGURE 2. Number of conference papers published by different libraries.

FIGURE 3. AI-ML driven digital twinning research statistics in different
fields.

Among the 850 articles that matched the designated key-
words, a total of 213 papers were selected after applying
the above inclusion and exclusion criteria. IEEE ACCESS
and Elsevier Journal of Manufacturing Systems are the top
two journals that have published the most articles within the
set criteria. The selected publications were first evaluated on
the basis of their titles and abstracts. The concept of digital
twinning in relation to the research questions was critically
examined, and a total of 63 papers were excluded in this
phase. Some paper-abstracts were not clear enough to be
directly evaluated, hence a full-text screening was performed
on 150 papers, resulting in the exclusion of 52 additional
papers. Snowball sampling was performed on the remaining
set of 98 papers. Then, we used the references and citations of
the selected papers to perform backward and forward search,
respectively, for identifying new potential papers.

Finally, a total of 117 papers concerning digital twin-
ning, its applications, and related technologies, were selected
for data extraction and synthesis of this study. Among the
117 articles, 61 articles discussed AI-ML based digital twins.
For each selected article, metadata forms were maintained
to categorize the information about the articles and to note
the observations assessed. The extracted metadata was then
coded for analysis, according to the year of publication,
authors’ names, affiliated universities or organizations, key-
words, name of journal or conference, research model, area
of focus, data source, and opportunities/issues highlighted.
The categories were derived according to the data needed to

32032 VOLUME 9, 2021

M. M. Rathore et al.: Role of AI, Machine Learning, and Big Data in Digital Twinning: A SLR, Challenges, and Opportunities

answer the research questions and for identifying the paper’s
main research areas. In addition to journal and conference
articles, we included 20 US patents, 15 technical web-reports,
and 5 standards, focusing on digital twinning. Some other
articles that indirectly relate to digital twinning, such as sup-
porting tools, technologies, and survey methodologies, are
also referred in our study.

III. DIGITAL TWIN: INTRODUCTION AND BACKGROUND
A. DIGITAL TWIN: DEFINITIONS AND CONCEPT
Researchers define digital twins in several ways. The pioneers
of digital twinning, Grieves and Vickers [17], define a digital
twin as ‘‘a set of virtual information constructs that fully
describes a potential or actual physical manufactured product
from the micro atomic level to the macro geometrical level.
At its optimum, any information that could be obtained from
inspecting a physical manufactured product can be obtained
from its Digital Twin.’’ In their opinion, the digital twin can
be any of the following three types: 1) digital twin prototype
(DTP); 2) digital twin instance (DTI); and 3) digital twin
aggregate (DTA). A DTP is a constructed digital model of
an object that has not yet been created in the physical world,
e.g., 3D modeling of a component. The primary purpose of a
DTP is to build an ideal product, covering all the important
requirements of the physical world. On the other hand, a DTI
is a virtual twin of an already existing object, focusing on only
one of its aspects. Finally, a DTA is an aggregate of multiple
DTIs that may be an exact digital copy of the physical twin.
For example, the digital twins of a spacecraft structure and a
spacecraft engine are considered DTIs that may be aggregated
into a DTA.

In this article, we assume the concepts of DTI and DTA
when referring to a DT. Note that, the majority of academic
scholars and industries follow similar definitions for a digital
twin. For instance, Glaessgen and Stargel [18] defined it from
the perspective of vehicles as ‘‘A digital twin is an inte-
grated multiphysics, multiscale, probabilistic simulation of
an as-built vehicle or system that uses the best available physi-
cal models, sensor updates, fleet history, etc., to mirror the life
of its corresponding flying twin.’’ Similarly, Tao et al. [19]
considered the aspect of product life cycle and interpreted
the digital twin as ‘‘a real mapping of all components in the
product life cycle using physical data, virtual data and inter-
action data between them.’’ Söderberg et al. [20] focused on
the application of optimization while defining a digital twin.
According to them, digital twinning is an approach to perform
a real-time optimization to a physical system using its digital
copy. Finally, Bacchiega [21] made it simpler by defining it
as ‘‘a real-time digital replica of a physical device.’’

With our understanding, shown in Fig. 4, digital twinning
is a process that involves the construction of 1) a cyber
twin that digitally projects a living or non-living physical
entity or a process (a system); and 2) a physical connection
between cyber and physical twins to share data (and informa-
tion) between them aimed at dynamic optimization, real-time
monitoring, fault diagnostics and early prediction, or health

FIGURE 4. Digital twinning concept.

monitoring of the physical counterpart. A physical twin can
be a process, a human, a place, a device, or any other object
with a special purpose, and which is able to be replicated in
the digital world as either a partial twin with limited function-
alities, or a complete twin that incorporates the full behavior
of its physical peer. Digital twinning is mostly employed
in industries for physical objects in their units. However,
there exist some digital twins that are mirrors of processes
in the physical world, such as digital twins of a mobile-edge
computing (MEC) system [22], human protein–protein inter-
action (PPI) [23], supply chain [24], components-assembly at
a manufacturing unit, and job scheduling [25].

B. A BRIEF HISTORY OF DIGITAL TWINS
The idea of creating a digital copy of a physical entity was
introduced in the early 2000s. However, the term ‘‘digital
twin’’ originated around ten years ago. Michael Grieves,
in one of his articles [2], claimed that the concept of dig-
ital twins was first presented during a lecture on product
life-cycle management (PLM) in 2003. Whereas, in his other
book chapter [1], he stated that the concept was originally
proposed, without a name, in 2002 while presenting a paper
in a special meeting at the University of Michigan Lurie
Engineering Center. Grieves mentioned in this book chapter,
‘‘While the name has changed over time, the concept and
model has remained the same.’’ He added that it was given
the name ‘‘mirrored spaces model (MSM)’’ in 2005 and
changed to ‘‘information mirroring model’’ in 2006. NASA
started using this concept of virtual and physical models in
their technology roadmaps [26] and proposals for sustain-
able space exploration [27] since 2010. However, the name
‘‘digital twin’’ was first coined in 2011 by John Vickers
of NASA. Practically, the first digital twin was developed
by Tuegel et al. [3] for the next-generation fighter aircraft,
in order to predict its structural life.

C. OPERATIONAL MECHANISM
Although the digital twin concept was introduced in 2002,
it became a popular trend due to the advancement in sensor
technology and IoT, which play a vital role in digital twinning
by collecting real-time data from the physical world and
sharing it with the digital world. The twinning can be viewed
as a bridge between a physical twin and the corresponding

VOLUME 9, 2021 32033

M. M. Rathore et al.: Role of AI, Machine Learning, and Big Data in Digital Twinning: A SLR, Challenges, and Opportunities

virtual twin. The physical-to-virtual connection is established
with a technology that allows the transfer of information from
the physical environment to its virtual twin, including web
services, cellular technology, WiFi, etc. The virtual twin is
adjusted gradually with the functioning of the physical twin
by continuously collecting the differences between the two
environments. These connections allow the monitoring of
responses to both conditions and interventions. The condi-
tions mainly occur in the physical environment, whereas the
interventions take place within the virtual twin. Thus, a digital
twin holds a real-time status of the physical counterpart.

The virtual-to-physical connections represent the
information circulating from the virtual to the physical envi-
ronment. This information may change the state of the physi-
cal twin by displaying some data or changing the system’s
parameters (for optimization, diagnostics, or prognostics).
Although virtual-to-physical connections are very helpful in
DT modeling, they are not always included in the description.
Instead, it is common to consider a one-way connection,
i.e., physical-to-virtual. Finally, the data and the information
from both physical and virtual worlds are stored and analyzed
at a centralized server—or a cloud computing platform—
where the final decisions related to optimization, diagnostics,
or prognostics, are made.

D. DIGITAL TWIN STANDARDS
Currently, there is no particular standard that solely focuses
on the technical aspects of digital twinning. Standardiza-
tion efforts are under-development by the joint advisory
group (JAG) of ISO and IEC on emerging technologies [28].
However, the ISO standard ISO/DIS 23247-1 [29] is the
only standard that offers limited information on digital twins.
In addition, there are other related standards that may facil-
itate DT creation. For example, the ISO 10303 STEP stan-
dard [30], the ISO 13399 standard [31], and the OPC unified
architecture (OPC UA) [32] technically describe ways to
share data between systems in a manufacturing environment.

IV. DIGITAL TWINNING IN INDUSTRIES: APPLICATIONS
Digital twinning is becoming apparent in various industries,
including manufacturing, medical, transportation, business,
education, and many more. In this section, we present the role
of digital twinning and the current research followed in these
areas.

A. MANUFACTURING
Digital twinning is conceived as a major tool in the man-
ufacturing industry to carry out smart manufacturing, fault
diagnosis, robotic assembly, quality monitoring, job shop
scheduling, and meticulousness management. In this way,
Rosen et al. [33] emphasizes the use of digital twinning in
manufacturing. Modules in a computerized system communi-
cate with each other during every step of the production, thus
depicting a realistic model of its physical counterpart. Simi-
larly, the work by Qi and Tao [8] explains the benefits of big
data-driven DT in smart manufacturing. The DT combines

all the manufacturing processes, starting from product design
to maintenance and repair. The virtual model is capable of
identifying the constraints of the virtual design in the physical
world, which are iteratively improved by the designers. Data
produced by sensors and IoT devices are then analyzed and
processed using big data analytics and AI applications to
enable the manufacturers to select a satisfactory plan.

On the other hand, DT is also used to monitor a component
or a product, considering its usage, health, and performance
during the life-cycle of manufacturing. Real-time data pro-
vided to the virtual model allows it to self-update and predict
any abnormal behaviors. Optimal solutions are developed for
problems found in the virtual models, and the actual manufac-
turing model is adjusted accordingly. Maintenance and repair
of the physical system can also be scheduled timely, based on
the predictions of the virtual models. One of such digital twin
projects is originated by Slovak University of Technology in
Bratislava [34] for a physical production line of pneumatic
cylinders, where they defined the continuous optimization of
production processes and performed proactive maintenance,
based on the real-time monitoring data. Similarly, a digital
twin of manufacturing execution system (MES) was devel-
oped by Negri et al. [35] that enables the supervisory con-
trol over the physical MES system using sensor technology,
by allowing the multi-directional communication between
digital and physical sides of manufacturing assets.

Several state-of-the-art works highlight that DTs should
be capable of self-healing and predictions. These predictions
play a vital role in an important aspect of smart manufactur-
ing, i.e., fault diagnosis, since a minor issue during production
can cause irreparable damages. A variety of technologies
used in fault diagnosis like Support Vector Machines [36],
Bayesian Networks [37], Deep Learning [38]–[40], and many
others [41]–[44] are capable of enhanced fault diagnosis.
However, Xu et al. [45] highlight that, in production sys-
tems, conditions are constantly changing. Therefore, the same
training model cannot be applied throughout the process, but
creating a new model requires a lot of time and resources.
As such, they proposed a digital twin-assisted fault diagno-
sis using deep transfer learning (DFDD) approach. DFDD
has been applied to fault diagnosis in smart and complex
manufacturing. The framework involves two phases. In the
first phase, the virtual model of the system is constructed.
Repeated designs of the model are tested and evaluated in the
virtual space until all anomalies are discovered. Simulation
data during design testing is provided to an embedded fault
diagnosis model in the virtual space. The diagnosis model
keeps learning from the simulation data using Deep Neu-
ral Networks, in order to increase its efficiency for failure
prediction during the start of the production phase when
there is insufficient training data. The second phase starts
when the virtual model achieves acceptable performance. The
physical entity is constructed and linked to its corresponding
virtual model. Data is transferred from a physical entity to the
virtual model through sensors during production. A diagnosis
model is formed and updated using the current data from

32034 VOLUME 9, 2021

M. M. Rathore et al.: Role of AI, Machine Learning, and Big Data in Digital Twinning: A SLR, Challenges, and Opportunities

the physical entity and the knowledge learned from the
previous phase, which is transferred using deep transfer
learning (DTL).

Robotic assembly, in industrial manufacturing, is responsi-
ble for handling a notable amount of work [46]. It is involved
in packaging, labeling, painting, welding, and many others.
With the advancements in the complexity of manufactur-
ing, these robotic assemblies have become more error-prone.
The concept of DT is being utilized in this area to monitor
and optimize the assembly process. In [47], a multisource
model-driven digital twin system (MSDTS) is designed for
robotic assembly. The MSDTS model consists of three parts.
The physical space consists of sensors, its associated data,
and the robotic arm for moving and gripping objects. The
virtual space consists of a server, a multisource model, and a
virtual reality display and control (VRDC). A communication
interface offers the exchange of data between two spaces in
real-time. Initially, a 3D model of the entire physical space
is constructed using a depth sensor that is mounted on the
robot arm. During operation, the VRDC provides a complete
view of the physical system by receiving a video stream from
an RGB camera. When the robot arm moves, angular data is
sent to the virtual twin through the communication interface
in real-time, and the graphical model in the virtual system
follows the same trajectory. The physical contact of the robot
arm with the surrounding object is simulated in the virtual
system using the Kelvin-Voigt model (KVM), where param-
eters of the model are estimated through the data of contact
force and relative motion of contact point. A surface-based
deformation algorithm is used to simulate the deformation
of an object using the data generated by KVM. The results
of the models are rendered in the VRDC. A complete view
of the system is provided to the operator via a head mount.
Interaction with the physical space is done using a control
handle.

Another important element in manufacturing is job shop
scheduling, which makes efficient use of resources to
reduce production time and maximize production effi-
ciency. In real-life situations, due to errors and anomalies,
the scheduling process can be rendered inefficient. With the
introduction of smart manufacturing and digital twins, new
DT-based job shop scheduling methods are introduced to
overcome scheduling plan deviation and provide a timely
response. One such model is proposed in [48]. A DT-based
job shop consists of a physical and a virtual space, which
communicate through a CPS. Scheduling data from the phys-
ical space is sent to the virtual space, and multiple scheduling
strategies are simulated and retrieved from the virtual models.
The finalized scheduling plan is fed into the physical space.
Since a physical system has many modules, the plan is divided
and categorized based on the respective modules. Continuous
communication between the physical and virtual space results
in achieving precise scheduling parameters, as well as pre-
diction of any disturbances in the schedule. The scheduling
plan can hence be updated and fed to the physical system for
increased efficiency and timely response.

Digital twin and big data are playing an important role
in smart manufacturing starting from product life-cycle to
maintenance and repair. Some of the stated research articles
highlighted the importance of digital twinning in the areas
of smart manufacturing. The concept of utilizing a variety of
data and integrating it with IoT, virtual reality, and data ana-
lytics, results in high fidelity monitoring, timely prediction
and diagnosis of faults in assembly or production, and overall
optimization and improvement of the manufacturing process.

B. MEDICAL
Applications of DT in medical include the maintenance of
medical devices and their performance optimization. DT,
along with AI applications, are also used to optimize the
life-cycle of hospitals by transforming a large amount of
patient data into useful information. The ultimate aim of the
digital twinning in healthcare is to help authorities in man-
aging and coordinating patients. Mater private hospitals in
Dublin (for cardiology and radiology) were facing problems
regarding increased services, patient demand, deteriorating
equipment, deficiency of beds, increased waiting time, and
queues. These problems indicated the call for the improve-
ment in the current infrastructure to cater to increasing
needs. Mater private hospitals (MPH) partnered with Siemens
Healthineers to develop an AI-based virtual model of their
radiology department and its operations [49]. As a result,
the simulations of the model provided insights towards the
optimization of workflows and layouts. The realistic 3D mod-
els of the radiology department, provided by DT techniques,
allowed for the prediction of operational scenarios and the
evaluation of the best possible alternatives to transform care
delivery.

In recent years, with the introduction of ‘‘precision
medicine,’’ the focus of DT technology is shifted towards a
human DT. Precision medicine is the branch of healthcare
that promotes tailored treatments on an individual level. The
human DT would be linked to its physical twin and would
display the processes inside the human body. It can result
in an easier and accurate prediction of illness with proper
context, and bring a paradigm shift in the way patients are
treated. Virtual physiological human (VPH) was the earliest
human DT that was developed [50]. VPHs would act as a
‘‘Virtual Human Laboratory’’ where each VPH was modified
based on the specific patient, and different treatments would
be tested on the modified VPH platform.

Apart from human DTs, organs or human body parts digital
twins have also been developed. Data from Fitbit devices,
smartphones, and IoT devices are sent in real-time to such
DTs, in order to provide constant feedback regarding human
organ activity. Some organs’ DTs have been used by experts
to perform clinical analysis, whereas many others are under
development. In a study, a 3D digital twin of a heart was
developed by Siemens Healthineers [51], after performing
a comprehensive research on approximately 250 million
images, functional reports, and data. The model exhibited
the physical and electrical structure of a human heart. This

VOLUME 9, 2021 32035

M. M. Rathore et al.: Role of AI, Machine Learning, and Big Data in Digital Twinning: A SLR, Challenges, and Opportunities

DT is currently under research at the Heidelberg university
hospital (HUH), Germany, where DTs of 100 patients have
been created, who had a history of heart diseases within a
period of six years. Simulations of these DTs were compared
with the ground truth, which provided promising results.

Another DT of the heart has been developed by researchers
at the Multimedia Communications Research Lab in Ottawa,
Canada. It is called a Cardio Twin and targets the detection
of ischemic heart disease (IHD) [52]. IHD is a condition
characterized by reduced blood flow to the heart, which can
lead to chest pain or mortality in case of delayed treatment.
The researchers developed the DT on the concept of edge
computing/analytics, where the time is considered very criti-
cal. Data is collected from social networks, sensors, and med-
ical records. The accumulated data is fed to an AI-inference
engine, where data fusion, formatting, and analytics are per-
formed using TensorFlow Lite to discover new information.
The Cardio Twin can communicate with the physical twin in
the real world, using a multimodal interaction component that
employs WiFi/4G or Bluetooth communication. Cardio Twin
performed a sample classification of 13420 ECG segments
with an accuracy of 85.77%, in a short span of 4.84 seconds.
However, no method to evaluate Cardio Twin in the real world
has been introduced.

Sim&Cure, a company based in Montpellier, France,
developed a simulation model for the treatment of aneurysm.
Aneurysm is an outward bulging of blood vessels, typically
caused by an abnormally weakened vessel wall. A serious
case of aneurysm can result in clotting, strokes, or death.
The last option for treating aneurysm is surgery. However,
endovascular repair (EVAR) is generally used, since it is
less invasive and low-risk. In EVAR, a stent-graft/catheter
is placed into the affected area to minimize the pressure.
In many cases, choosing the stent-graft/catheter is difficult
and depends on the size of the blood vessels. The Sim&Cure’s
DT helps surgeons in selecting an ideal implant to cater to the
size of the aneurysm as well as the blood vessels. A 3D model
of the affected area and surrounding vessels is created, and
multiple simulations are run on the personalized DT, which
allows surgeons to have a better picture. Promising results
have been presented in preliminary trials [53], [54].

Researchers at the Oklahoma State University developed a
human airway DT—named ‘‘virtual human’’—in their com-
putational biofluidics and biomechanics laboratory (CBBL).
They tracked the flow of air particles in aerosol-delivered
chemotherapy and found out that, the aerosol-based drug
hit the cancerous cells with less than 25% accuracy [55].
This caused more harm than benefits to patients, as the
remaining drug would fall on healthy tissue. The version
1.0 of ‘‘virtual human’’ was based on a 47-year-old standing
male, containing the entire respiratory system. V1.0 also
allowed patient-specific structural modifications, e.g., creat-
ing a respiratory system of a standing/seated female or a kid
with/without respiratory conditions. Following the success of
V1.0, CBBL researchers developed its successor version 2.0.
The V2.0 was patient-specific, and was created by performing

an MRI/CT scan of the patient. The scanned data was used
to construct a 3D model of the lungs. The researchers at
CBBL then created a virtual population group (VPG), which
was a large group of human DTs. The VPG exhibited trends
within different groups/sub-groups. Simulations to analyze
the trends of aerosol particle movement were conducted on
the VPG, by varying the particle sizes, inhalation rate, and
initial position of the medication. These simulations indicated
that the drug’s effectiveness would increase to 90% if the drug
delivery method was personalized to each patient, rather than
distributing the drug evenly for every patient [55].

In another study, Liu et al. [56] proposed a cloud-based
DT healthcare solution (CloudDTH) for elderly people. The
cloud-based solution provides a fusion of physical and virtual
systems to address real-time interaction between patients
and medical institutions, and personalized healthcare for the
entire life-cycle of the elderly. CloudDTH has a layered archi-
tecture, providing health resources, identification of medical
personnel, user interface, virtualization, and security services
to users. CloudDTH obtains real-time data from sensors for
ECG, BP, pulse rate, and body temperature. These sensors
are already implemented in the CloudDTH framework. The
sensor data are then transmitted to the cloud server, using
TCP. In case of an incident, such as patient falling, heart
attack, stroke, etc., the monitoring model, after performing
analysis on the received data, sends a high-frequency and
multi-attribute monitoring order of the patient to medical
personnel. A case study was performed by researchers, where
data from two patients with normal and abnormal heart rates
was input to the system. The simulation results indicated
symptoms of arrhythmia in one patient, and recommended
the dosage of medication based on their physical conditions.
The CloudDTH framework simulations also provided a fea-
sible scheduling mechanism for elderly patients in hospitals,
in order to avoid long queues.

C. TRANSPORTATION
Numerous innovative technologies have been brought for-
ward with the development of IoT, including digital twins,
autonomous things, immersive technology, etc. Various types
of digital twins are developed in transportation sector, includ-
ing DTs for automobile components, vehicles, vehicular
networks, and road infrastructures. However, the purpose
remains the same i.e., monitoring, optimization, and prognos-
tics and health management. For example, Wang et al. [57]
developed a framework for connected vehicles based on
digital twins. The framework used vehicle-to-cloud (V2C)
communication to provide advisory speed assistance (ADSA)
to the driver. Real-time data from sensors was obtained in
the physical system, which was sent to the cloud through
the V2C module. All processing of the data from V2C was
performed on the cloud server. The computed results were
sent back to the physical system and served as a guidance sys-
tem for components within the physical world. The authors
demonstrated the effectiveness of their framework with a case
study of cooperative ramp merging involving three passenger

32036 VOLUME 9, 2021

M. M. Rathore et al.: Role of AI, Machine Learning, and Big Data in Digital Twinning: A SLR, Challenges, and Opportunities

vehicles, and the results showed that the digital twin can
indeed assist transportation systems.

Cioroaica et al. [58] worked on the context of connected
vehicles in smart ecosystems. The establishment and achieve-
ment of goals in smart ecosystems are possible when smart
entities within the ecosystem co-operate with each other.
This is achieved when the systems have a level of trust.
The authors developed a virtual hardware-in-the-loop (vHiL)
testbed model to evaluate the trust-building capability of
smart systems within an ecosystem. A smart agent, capa-
ble of interacting with the vehicle’s electronic control unit
(ECU), is installed at the vehicle along with its corresponding
DT. In Phase 1, the trustworthiness of the smart agent is
evaluated by simulation in its corresponding virtual twin.
Phase 2 involves trust-building, where the smart agent is
executed on the ECU. Evaluation of simulated and actual
results identifies the obstacles. These obstacles are overcome
by the collaboration of virtual and physical entities to achieve
trustworthiness in a smart ecosystem.

Chen et al. [59] studied the use of unmanned aerial vehi-
cles (UAVs) as complementary computation resources in a
mobile edge computational (MEC) environment for mobile
users (MU). MEC provides computational capabilities to
MUs within a radio access network (RAN). Mobile users send
computational tasks to UAVs by creating the corresponding
VMs. The tasks arriving at the UAVs are stored in queues and,
due to limited resources, the MUs have to compete for them.
The authors proposed deep reinforcement learning (DRL)
techniques for the scheduling of tasks on the UAV, and for
minimizing the response delay from the UAV to the MUs.
The training of the DRL network in an offline manner is
achieved by creating a digital twin of the entire MEC system.
Simulations with varying parameters were conducted and the
best results were selected. The results of the DRL scheme
trained on digital twins ensured significant performance gains
when compared to other baseline approaches.

Digital twins have also been utilized in transportation sys-
tems for traffic congestion management, congestion predic-
tion, and avoidance. Kumar et al. [60] worked on intelligent
transport systems, leveraging technologies such as fog/edge
analytics, digital twins, machine learning, data lakes, and
blockchain. The authors captured situational information
from cameras, and performed edge analytics on the acquired
data. An entire virtual vehicle model was created via a dig-
ital twin to replicate the real-world scenario. Driver inten-
tions were predicted using machine and deep learning algo-
rithms to avoid traffic congestion. This virtual vehicle model
allowed autonomous vehicles to make decisions regarding
optimal paths, but also helped drivers of non-autonomous
vehicles to make better decisions based on the traffic situation
and the mined driver intentions.

Digital twins have also been used for the maintenance
of different systems. The work implemented by Venkate-
san et al. [61] monitored and projected the health conditions
of electric motor vehicles using an intelligent digital twin
(i-DT). The framework tracked the health of the electric

motor in an electric vehicle using fuzzy logic and artificial
neural networks (ANNs). The average speed of the vehicle
and the duration of travel was fed into the ANN i-DT and
fuzzy logic i-DT for training purposes. In addition, simula-
tions carried out on a digital twin tested the performance of
the entire framework. Parameters such as winding and casing
temperature, deterioration in magnetic flux, and lubricant
refill time were set for the digital twin. The comparison of
theoretical and i-DT computations indicated that an i-DT can
effectively be used in electric vehicles to foresee their health.

D. EDUCATION
Another important area where digital twins can play a crucial
role, is education. Digital twins of physical entities such
as labs, construction, mechanical equipment, can be created
and provided to students for online learning. However, there
has not been a lot of research effort on the use of DT in
the education domain. One such work was performed by
Sepasgozar [62] that used digital twins and virtual gaming
for online education. The authors created a digital twin of
an excavator along with a virtual game for the course of
construction management and engineering. The project con-
tained four modules named 1) group wiki project and role
play (GWiP); 2) interactive construction tour 360 (ICRT 360);
3) virtual tunnel boring machine (VTBM); and 4) piling
augmented reality and digital twin (PAR-DT). GWiP was
used for doing group projects online. ICRT 360 consisted of
recorded videos to provide details on construction sites and
machinery. VTBM was a virtual game-based environment
that helped students to learn about the working of a tunnel
boring machine. Virtual equipment was introduced in the
game, where a student or a group of students could explore
their interests. PAR was developed for smartphones and Ocu-
lus headsets to provide students an augmented environment
to collaborate and understand the importance of piling in
construction. The final module involved a digital twin of
an excavator, which was also linked to a physical instance.
The DT provided hands-on learning about the functions and
movements of an excavator. The students’ feedback empha-
sized the importance of an immersive environment in online
education.

E. BUSINESS
Business is also one of the areas where DT is playing an
important role. According to PropTechNL [63], the real estate
sector is fragmented in terms of architects, installation, con-
struction, transport, and management. This fragmentation
results in an inefficient system that has a negative impact
on people living in a society. Digital twins can provide huge
opportunities in real state, and facilitate the creation of smart
societies. For example, a wide range of sensors can collect
data, and the performance of a building can be measured and
improved. Digital twins in real estate may add significant
value by re-positioning buildings to the needs and require-
ments of customers, hence improving the customer experi-
ence. The design of buildings, the usage, effectiveness, and

VOLUME 9, 2021 32037

M. M. Rathore et al.: Role of AI, Machine Learning, and Big Data in Digital Twinning: A SLR, Challenges, and Opportunities

strength of raw materials, as well as maintenance and running
costs, can be managed through digital twins. Thus, it provides
a cost-effective, fast, and smart way of developing a real
estate. For instance, an American multinational company,
GE Healthcare, has incorporated the use of DT to redesign
its systems, in order to run new hospitals more efficiently.

Kampker et al. [64] introduced a framework for the devel-
opment of successful business models in smart services. The
scenario of crop (potato) harvesting was taken into consider-
ation during their research. In traditional harvesting mecha-
nisms, the harvesting machines are set up based on historical
data and the experiences of individual operators. However,
the lack of standard procedures may cause damage to the
crop. Therefore, the authors developed a framework, based
on a digital twin, to reduce the damage to the crop during
harvesting. Specifically, a digital twin is set up near the phys-
ical field. The virtual model then passes through the same
stages as the real crop. During the simulation, the condition
of the neighboring crop is analyzed for potential damage.
The results of the analysis lead to adjusting the parameters,
and repeated simulations continue until the optimal settings
are found. Tests carried out by the authors indicated that
more damage to the crop is caused by its impact on multiple
conveyor belts during the transition. Hence, adjustment to
the height and position of conveyor belts can reduce the
risk of damage. This framework can also tweak the settings
of autonomous harvesting machines, apart from providing
recommendations to operators.

F. OTHER INDUSTRIES
Digital twinning can be a part of smart construction, where
a DT may be designed for buildings, roads, or any other
infrastructure development. For example, a virtual twin was
developed for office buildings [65] that manages the build-
ing’s life-cycle, by collecting data through sensors. Further-
more, DT technology may advance the disaster management
approaches in smart cities [66]. Possibly, the technology also
has a potential to protect industrial control systems and data
from cyber attacks. On this account, Dietz and Pernul [67]
proposed the use of digital twinning technology to identify
security threats that target industrial control systems (ICSs),
and rectify their effects. Theoretically, they focused on the
Stuxnet worm [68] that compromised the speed of centrifuge,
and Triton [69] that digitally invaded a petrochemical plant
in Saudi Arabia. Deitz et al., indicated in the Stuxnet exam-
ple that the outliers in the historical network traffic would
have detected a threat. Similarly, in the case of simulations,
the deviation of network traffic between the virtual and phys-
ical systems would have identified the attack.

V. AI-ML AND BIG DATA: AN INTRODUCTION
Big data remains one of the top research trends from last
few years. It is different from an ordinary data because of
its high volume, high velocity, and heterogeneous variety,
as interpreted in Fig. 5. Researchers named these character-
istics as ‘‘the 3Vs of big data,’’ i.e., volume, velocity, and

FIGURE 5. Big data definition.

variety. Later, two more Vs—value and veracity—were added
to the list. Thus, we refer to any data as big data, if it is
of significant size (volume), it is being produced at very
high-speed (velocity), and it is heterogeneous with structured,
semi-structured, or unstructured nature (variety). The worth
of big data analytics brings the fourth V (i.e., value) into
its characteristics, thus making it an asset to the organiza-
tion. Big data analytics is a process that analyzes big data
and converts it to valuable information, using state-of-the-art
mathematical, statistical, probabilistic, or artificial intelli-
gence models. However, the 3Vs of big data lead us to a new
world of challenges, including capturing, storing, sharing,
managing, processing, analyzing, and visualizing such high-
volume, high-velocity, and diverse variety of data. To this end,
various frameworks [70]–[73] have been designed to handle
big data for effective analytics in different applications.

Artificial intelligence (AI) is the digital replication of
three human cognitive skills: learning, reasoning, and self-
correction. Digital learning is a collection of rules, imple-
mented as a computer algorithm, which converts the real
historical data into actionable information. Digital reasoning
focuses on choosing the right rules to reach a desired goal.
Whereas, digital self-correction is the iterative process of
adopting the outcomes of learning and reasoning. Every AI
model follows this process to build a smart system, which
performs a task that normally requires human intelligence.
Most of the AI systems are driven by machine learning, deep
learning, data mining, or rule-based algorithms, where others
follow logic-based and knowledge-based methods. Nowa-
days, machine learning and deep learning are widely used AI
approaches.

It is often confusing to differentiate between artificial
intelligence, machine learning, and deep learning techniques.
Machine learning is an AI method, which searches for partic-
ular patterns in historical data to facilitate decision-making.
The more data we collect, the more accurate is the learning
process (reflects the value of big data). Machine learning
can be 1) supervised learning, which accepts data sets with

32038 VOLUME 9, 2021

M. M. Rathore et al.: Role of AI, Machine Learning, and Big Data in Digital Twinning: A SLR, Challenges, and Opportunities

labeled outputs in order to train a model for classification or
future predictions; 2) unsupervised learning, which works on
unlabeled data sets and is used for clustering or grouping; and
3) reinforcement learning, which accepts data records with no
labels but, after performing certain actions, it provides feed-
back to the AI system. Examples of supervised learning tech-
niques are regression, decision trees, support vector machines
(SVMs), naive Bayes classifiers, and random forests. Sim-
ilarly, K-means and hierarchical clustering, as well as mix-
ture models, are examples of unsupervised learning. Finally,
Monte Carlo learning and Q-learning fall under the reinforce-
ment learning category. On the other hand, deep learning is a
machine learning technique that is motivated by biological
neural networks with one or more hidden layers of digital
neurons. During the learning process, the historical data are
processed iteratively by different layers, making connections,
and constantly weighing the neuron inputs for optimal results.
In this article, we mainly focus on digital twin systems based
on machine learning.

VI. RELATIONSHIP BETWEEN IoT, BIG DATA, AI-ML,
AND DIGITAL TWINS
The emerging sensor technologies and IoT deployments in
industrial environments have paved the way for several inter-
esting applications, such as real-time monitoring of phys-
ical devices [74], indoor asset tracking [75], and outdoor
asset tracking [76]. IoT devices facilitate the real-time data
collection—that is necessary for the creation of a digital twin
of the physical component [77], [78]—and enable the opti-
mization [79] and maintenance [80] of the physical compo-
nent by linking the physical environment to its virtual image
(using sensors and actuators). Note that, the above-mentioned
IoT data is big in nature [81] (as explained in Section V),
so the big data analytics can play a key role in the develop-
ment of a successful digital twin. The reason is that indus-
trial processes are very complex, and identifying potential
issues in early stages is cumbersome, if we use traditional
techniques. On the other hand, such issues can easily be
extracted from the collected data, which brings efficiency
and intelligence into the industrial processes. However, han-
dling this enormous amount of data in the industrial and DT
domains requires advanced techniques, architectures, frame-
works, tools, and algorithms. For instance, Zhang et al. [82],
[83] proposed a big data processing framework for smart
manufacturing and maintenance in a DT environment.

Oftentimes, cloud computing is the best platform for pro-
cessing and analyzing big data [84]. Additionally, an intelli-
gent DT system can only be developed by applying advanced
AI techniques on the collected data. To this end, intelligence
is achieved by allowing the DT to detect (e.g., best pro-
cess strategy, best resource allocation, safety detection, fault
detection) [85], predict (e.g., health status and early main-
tenance) [80], [86], optimize (e.g., planning, process con-
trol, scheduler, assembly line) [87], [88], and take decisions
dynamically based on physical sensor data and/or virtual twin
data. In short, IoT is used to harvest big data from the physical

FIGURE 6. Relationship between IoT, big data, AI-ML, and digital twins.

environment. Later, the data is fed to an AI model for the
creation of a digital twin. Then, the developed DT can be
employed to optimize other processes in the industry. The
overall relationship among IoT, big data, AI, and digital twins
is presented in Fig. 6.

VII. CURRENT DEPLOYMENTS OF DIGITAL TWINS USING
BIG DATA AND MACHINE LEARNING
We have identified the primary sectors where DT-based sys-
tems are developed with the help of AI-ML techniques. In the
following sections, we discuss the current deployments in
these sectors, including smart manufacturing, prognostics
and health management (PHM), power and energy, automo-
tive and transport, healthcare, communication and networks,
smart cities, and others.

A. SMART MANUFACTURING
Smart manufacturing involves 1) the acquisition of data from
manufacturing cells through a variety of sensors; 2) the
management of the acquired data; and 3) the data exchange
between different devices and servers. In a DT environment,
the data is collected from a physical manufacturing cell and/or
its corresponding virtual cell. Such data can be further utilized
for manufacturing process optimization, efficient assembly
line, fault diagnosis, etc., using AI approaches. The AI-ML
based digital twinning process for smart manufacturing is
depicted in Fig. 7.

Manufacturing is the top industry where most digital twins
are being developed. Xia et al. [91] proposed a manufacturing
cell digital twin to optimize the dynamic scheduler for smart
manufacturing. An intelligent scheduler agent, called digital
engine, was developed and trained for optimization using
deep reinforcement learning algorithms (DRLs), such as nat-
ural deep Q-learning [101], double deep Q-learning [102],
and prioritized experience replay (PER) [103]. The underly-
ing features were captured from both the physical and virtual
environments of the cell by an open platform communica-
tions (OPC) server. The training of the DRL network was
done through a gradient descent process, which requires finite
learning iterations and is sufficiently intelligent, reliable, and
robust. The developed DT-based dynamic scheduler opti-
mizes the manufacturing process by accelerating the training,
testing, and validation of smart control systems. The system
was tested on a robot cell to optimally select the strategy

VOLUME 9, 2021 32039

M. M. Rathore et al.: Role of AI, Machine Learning, and Big Data in Digital Twinning: A SLR, Challenges, and Opportunities

FIGURE 7. DT-based smart manufacturing using big data analytics and
AI-ML.

for performing the lower level tasks that are necessary to
accomplish the higher level manufacturing goal.

Zhou et al. [79] performed a geometric optimization
of centrifugal impeller (CI) by collecting features, such
as meridional section (MS), straight generatrix vectors
(SGV), and set of streamlines (SSL), from both the physi-
cal and CAD-based digital model of the CI. However, with
the improvement in machinability, the DT-based geometric
optimization reduces the aerodynamic performance. Thus,
the best model for the CI is selected by training the deep
deterministic policy gradient (DDPG) reinforcement learn-
ing model [104] to iteratively select the fair geometry of
the CI-design with optimum values of machinability and
aerodynamic performance. For the DDPG algorithm, they
used two actor networks (online and target network) as the
strategy functionπ to control the agent-actions, and two critic
networks (online and target network) to evaluate these actions
and give rewards. The proposed DT-based optimization sped
up significantly the design and manufacturing of the impeller.
Similarly, Zhang et al. [95] also developed an impeller DT,
but for the purpose of manufacturing process planning. They
employed a knowledge reuse deep learning network (PKR-
Net) [105], which takes data from dynamic knowledge base,
views from 3D computer-aided impeller design (CAD), 2D
drawings, and process knowledge. The objective is to opti-
mize the theoretical processes and generate the best process
plan for product manufacturing, by considering both manu-
facturing time and monetary cost.

Furthermore, Lee et al. [106] designed a deep learning and
cyber-physical system based digital twinning (DTDL-CPS)
architecture for smart manufacturing, that can be used in shop
floor optimization, fault diagnosis, product design optimiza-
tion, and predictive maintenance. BDHDTPREMfg [84] is
a similar CPS-based big data-driven model for DT-enabled
re-manufacturing. Several other digital twins have been

developed in the manufacturing industry using AI approaches
that could not be fully discussed in this article. Rather,
Table 2 summarizes these digital twins with respect to the
problem they solved (i.e., the application), the ML-approach
they used to solve the problem, and the DT use-case they
developed.

B. PROGNOSTICS AND HEALTH MANAGEMENT
The persistent use of a product degrades its performance over
time, which may lead to malfunctioning. Thus, prognostics
and health management (PHM) is very crucial in all indus-
tries. PHM process involves the prediction of the remaining
useful life of a product and the consistent monitoring of its
health. This is the second most important application of DT,
following smart manufacturing. Note that, several alternative
terms, such as ‘‘predictive modeling’’ [86], ‘‘structural life
prediction’’ [3], ‘‘remaining useful life’’, and ‘‘predict and
act’’ [107] have also been used in place of PHM. DT-based
PHM regularly monitors the physical equipment based on
the data generated by the equipment-sensors, performs diag-
nosis and prognosis operations on the data using big data
analytics and AI, and recommends design rules for immediate
maintenance. The process of DT-enabled PHM is depicted
in Fig. 7.

Tao et al. [108] developed a digital twin for a wind turbine
in a power plant, in order to monitor its health by perform-
ing gearbox prognosis and fault detection. The proposed
DT-driven PHM can be applied to any complex equipment in
harsh environments, such as aircraft, ships, and wind turbines.
The wind turbine DT is built based on various geometry
levels, physics, behavior, and rules. The DT can detect the
disturbances in the turbine environment, as well as potential
faults in itself and its designed model. The data is collected
from the DT model (both physical and digital) and is matched
against the thresholds for degradation detection. In addition,
past DT-data is used to train a single hidden layer neural
network for better prediction of gradual faults and detection
of its causes, using extreme learning machine (ELM) [109].
The abrupt fault in the turbine is detected by comparing the
data from the physical and virtual environments. Similarly,
to improve ship efficiency and avoid unnecessary mainte-
nance operations, a data-driven ship digital twin was devel-
oped by Coraddu et al. [110]. Their goal was to determine the
speed loss due to marine fouling. Multilayered-deep extreme
learning (DELM) [111] predicts the ship’s speed, based on the
features collected from on-board sensors, such as designed
and ground speed, draft, engine and shaft generator power,
wind speed, temperature, fuel consumption, etc. The expected
ship speed is compared with the measured speed to compute
the speed loss. Finally, robust linear regression is applied to
the speed loss information to determine whether the speed
loss is due to marine fouling.

Numerous other digital twins have been developed
for PHM of industrial components, including pho-
tovoltaic energy conversion unit [112], battery sys-
tem [113], vehicle motor [61], UAV [115], spacecraft [116],

32040 VOLUME 9, 2021

M. M. Rathore et al.: Role of AI, Machine Learning, and Big Data in Digital Twinning: A SLR, Challenges, and Opportunities

TABLE 2. State-of-the-art AI-ML developments in digital twinning for smart manufacturing.

aircraft [3], [118]–[120], gillnet [122], gearbox, aircraft-
turbofan engine, rotating shaft-bearing [121], etc. All these
systems are summarized in Table 3.

C. POWER AND ENERGY
In the power and energy sector, most of the DTs are developed
in electronic systems, wind-power farms, cooling systems,
and fuel-related systems. The digital twin of an inverter
model [125] was developed by imitating the voltage con-
troller, the current control loop, and the controlled plant,
based on three distinct neural networks (NNs). Each of
the three NNs is trained on real data collected from the
physical model, where the back propagation (BP) algorithm
is deployed to tune, in real-time, the proportional–integral
(PI) controller. Also, Andryushkevich et al. [126] introduced
the digital twin of power-system using ontological model-
ing. The developed DT selects the optimal configuration
of the hybrid power supply system, by utilizing genetic
algorithms [127]. Likewise, a digital twin framework for
power grids was designed by Zhou et al. [128] to per-
form real-time analysis. Specifically, NN-based learning was
applied to predict the grid operational behavior for fast secu-
rity assessment, based on the voltage stability and oscillation
damping.

In addition, a DT for a dew-point cooler was devel-
oped [99] to improve its cooling performance, by optimizing
operational and design parameters, including cooling capac-
ity, coefficient of performance (COP), dew point efficiency,
wet-bulb efficiency, supply air temperature, and surface area.
The DT of the cooler is developed with feed-forward neural
networks (FFNNs), and digitally mimics the cooler’s behav-
ior by utilizing the air characteristics (i.e., temperature, rel-
ative humidity) as well as the main operational and design
parameters (i.e., air velocity, air fraction, HMX height, chan-
nel gap) as inputs to the FFNN. Later, the DT-collected data
are supplied to a genetic algorithm (GA) for multi-objective
evolutionary optimization, in order to maximize cooling,
COP, and wet-bulb efficiency, and minimize the surface area
within four diverse climates (i.e., tropical rainforest, arid,
Mediterranean hot summer, and hot summer continental cli-
mates).

Apart from design and performance optimizations,
ML-based PHM is accomplished for power and energy
related components with the use of DTs, such as wind-
turbine, [108], electric vehicle motor [61], photovoltaic sys-
tems [112], battery systems [113], plasma radiation detection
in metal absorber–metal resistor bolometer [114], as dis-
cussed in Section VII-B.

VOLUME 9, 2021 32041

M. M. Rathore et al.: Role of AI, Machine Learning, and Big Data in Digital Twinning: A SLR, Challenges, and Opportunities

TABLE 3. State-of-the-art AI-ML research in industrial digital twinning for PHM.

D. VEHICLES AND TRANSPORTATION
A vehicle digital twin was developed by Alam and El
Saddik [85] in a vehicular cyber-physical system (VCPS),
by mimicking its speed behavior, fuel consumption, and
airbag status. The system utilized fuzzy rule base with a
Bayesian network [129], in order to build a reconfiguration
model for driving assistance. Similarly, Kumar et al. [60]
built virtual models of running vehicles in the cloud, which
obtained real-time road and vehicular data through fog or
edge devices, in order to avoid traffic congestion. The driver
behavior and intention are predicted using machine learning
on historical data. LSTM-based recurrent neural networks
(RNNs) [130] are applied on the data to obtain the best
route for a particular vehicle. Besides, digital twins have
also been developed for vehicular network system, itself.
For instance, the digital twin of a mobile edge comput-
ing (MEC) system was developed [59] for resource alloca-
tion in unmanned aerial vehicle (UAV) networks, using deep
recurrent Q-networks (DRQNs) [131]. Likewise, the digital
twin of software-defined vehicular networks (SDVNs) [132]
allows for the predictive verification and maintenance diag-
nosis of running vehicles network, using machine learning.
Furthermore, prognostics and health management is con-
ducted by developing digital twin of aircraft [118] and space-
craft [116], ship [110], and electric vehicle motor [61]. All of
these PHM approaches employ machine learning techniques.

E. HEALTHCARE
In healthcare, the majority of AI-ML enabled DTs are human
digital twins [23], [56], [133]–[136]. Mimicking the full
functionalities of a human body is not currently possible,
thus, a human digital twin can only focus on limited aspects
of human biology. For example, the digital twin by Barri-
celli et al. [133] focuses on fitness-related measurements of
athletes. Specifically, their virtual patient classified physi-
cal athletes and predicted their behavior using KNN classi-
fiers [137] and support vector networks [138], by training
models on physical patient data collected by IoT devices.
Björnsson et al. [23] concentrated on protein–protein inter-
action (PPI) networks to diagnose and treat patients of a
particular disease. Their model is implemented as an AI
system that monitors the effect of drugs on the human body,
using machine learning tools, such as Bayesian networks,
deep learning, and decision trees.

Furthermore, Chakshu et al. [135] mimicked the patient’s
head behavior for detecting the severity of carotid stenosis.
Their model selects components from a patient video and
applies principal component analysis (PCA) to identify the
severity of carotid stenosis, by comparing it with the virtual
model components. The authors also recommended the use of
deep learning, machine learning, and other AI techniques for
better detection accuracy. Similarly, Mazumder et al. [134]
digitally replicated the process of generating synthetic PPG

32042 VOLUME 9, 2021

M. M. Rathore et al.: Role of AI, Machine Learning, and Big Data in Digital Twinning: A SLR, Challenges, and Opportunities

signals to create the digital twin of a cardiovascular sys-
tem. In the virtual model, parameters are optimized using a
particle-swarm-optimization (PSO) algorithm. The algorithm
minimizes the integral-squared-error (ISE) in the feature set,
in order to generate the synthetic PPG signal. On the other
hand, Laamarti et al. [136] and Liu et al.’s [56] models are
generic ML-enabled frameworks for providing health ser-
vices to elderly people.

F. COMMUNICATIONS AND NETWORKS
In the networking and communications domain, the digi-
tal twin of an indoor space environment [139] is imple-
mented to model, predict, and control the terahertz (THz)
signal propagation characteristics in an indoor space. The
DT selects the best THz signal path from the base station
to the mobile target, by avoiding obstacles. The DT iden-
tifies the obstacle, its position, and dimensions, by apply-
ing a you only look once (YOLO) machine learning algo-
rithm [140] on the monochromatic image of the obstacle.
Furthermore, deep learning algorithms are used for material
texture recognition and classification. On the other hand,
a new network architecture, equipped with ML-based virtual
twin of a software-defined vehicular network (SDVN) [132],
is designed to benefit from intelligent networking and adap-
tive routing. Dong et al. [22] developed a similar digital twin
of a real network for mobile edge computing. The virtual
model of the MEC is equipped with a deep neural network
that is frequently updated based on the variation of the real
network. The model then selects the optimal resource alloca-
tion and offloading policy at each access point.

G. SMART CITIES
In the smart city sector, a Zurich city digital twin [141]
was developed by transforming 3D spatial data and city
models, including buildings, bridges, vegetation, etc., to a
virtual world. The authors discussed the effects of urban cli-
mate, which can be predicted by machine learning techniques
based on the current weather and air-quality data. Similarly,
a Vienna city digital geoTwin [142] can be linked with city
data, such as socioeconomic, energy consumption, and main-
tenance management data, in order to make it a living digital
twin with the aim of AI technologies. Furthermore, a vision
for integrating artificial and human intelligence for a disaster
city digital twin is introduced by Fan et al. [66]. Finally,
a geospatial digital twin [143] is the digital replica of a spatial
entity, where machine learning and deep learning techniques
are used for interpretation, analysis, and organization of 3D
point clouds.

H. OTHER INDUSTRIES
DT systems that utilize AI-ML techniques have been
deployed in other industries as well. For instance, the supply
chain DT by Marmolejo-Saucedo [24] was developed for a
pharmaceutical company, using machine learning and pattern
recognition algorithms. The objective was to identify the
behavior, dynamics, and changing trends in the supply chain.

Data management for DT environments is another area of
active research. Specifically, a DT-enabled collaborative data
management framework was proposed, using edge and cloud
computing [100]. The goal was to perform advanced data
analytics in additive manufacturing (AM) systems, in order to
reduce the development time and cost, and improve the prod-
uct quality and production efficiency. To this end, the authors
introduced cloud-DTs and edge-DTs, developed at different
product life-cycle stages, which communicate with each other
in order to support intelligent process monitoring, control,
and optimization. As a use case, the framework was imple-
mented within the MANUELA project, where layer defect
analysis was performed by a deep learning model on product
life-cycle data. Moreover, Tong et al. [144] introduced an
intelligent machine tool (IMT) digital twin model for machin-
ing data acquisition and processing, using data fusion and ML
approaches.

VIII. DATA-DRIVEN DIGITAL TWINNING PATENTS
The importance of DT technologies can be verified by the
number of patents in this field. In particular, more than one
thousand patents have been awarded on AI-enabled digi-
tal twinning in all around the world. A wind-power farm
digital twin was filed as a U.S. Patent in 2016 by Gen-
eral Electric (GE) [145], where the DT is composed of two
communication networks: 1) a farm-based communication
network, which enables the coupling of control systems from
individual wind turbines with the main wind farm control
system and with other wind turbines; and 2) a cloud-based
communication network that is composed of an infrastruc-
ture of digital wind-turbine models, where the plurality of
the models are continuously changing during farm opera-
tion, by investigating and analyzing data generated by the
farm-based communication network using machine learning.
Furthermore, they provided a fully functional graphical user
interface (GUI) of the digital wind-turbines, where the user
can control the input features of the DT model to optimize
the performance of the wind farm using machine learning
algorithms. In another patent, Shah et al. [146] developed the
digital twin of a vehicle cooling system, by using status data
(such as health scores) to predict cooling system failures and
optimize its performance. Similar data-driven digital twin-
ning systems have been designed in the energy and power
sector [147].

In predictive analytics for machine maintenance, GE’s Her-
shey et al. [148] developed a system to predict the lifetime of
a component in the electromechanical industry (such as an
aircraft engine), by developing a digital twin of the physical
system. The component is monitored by IoT-based sensors
and its remaining life is assessed based on the monitoring
conditions. In this process, they developed a stress analysis
model, a fluid dynamics model, a structural dynamic model,
a thermodynamic model, and a fatigue cracking model. Then,
they utilized probabilistic models, such as a Kalman filter,
to predict the lifetime and detect component faults. Sim-
ilarly, the Siemens corporation designed a generic digital

VOLUME 9, 2021 32043

M. M. Rathore et al.: Role of AI, Machine Learning, and Big Data in Digital Twinning: A SLR, Challenges, and Opportunities

twin model [149] for a variety of machines, including heat-
ing, ventilation, and air conditioning (HVAC). They uti-
lized data-driven approaches for energy-efficient machine
maintenance, utilizing sensor data and model-based analyt-
ics. Several other patents focus on predictive analytics with
AI-enabled digital twins [150]–[152].

A few digital twin patents have also been developed in the
healthcare sector. GE researchers designed a patient DT [153]
to diagnose diseases, treat, and prescribe medicines. The
digital representation of the patient (i.e., the DT) consists of
medical record data structures, medical images, and historical
patient information. The DT is equipped with healthcare soft-
ware applications (such as expert systems), patient medical
data, and AI models (neural networks, machine learning) that
can diagnose, identify health issues, and prescribe treatments
(e.g., medication, surgery, etc.). Also, Nagesh [154] build
an X-ray tube DT to predict tube-liquid bearing failures.
He used X-ray tube housing vibration data, collected by a
sensor in a free run mode of an X-ray tube, and applied
AI-based prediction. There are also patents in DT-based
surgery for the healthcare industry that utilize data-driven
approaches [155], [156].

Finally, there are hundreds of additional patents that
emphasize AI-enabled data-driven digital twinning, which
could not be covered here. These digital twinning systems
belong to a variety of industrial sectors, including manufac-
turing [157], [158], run-time environment [159], transport
and automotive industry [160]–[162], building and construc-
tion systems [163], etc.

IX. EVALUATING A SUCCESSFUL DIGITAL TWIN
A successful digital twin can only be justified when its virtual
twin closely matches the functionality of its physical coun-
terpart. This justification can be presented by comparing the
outputs of the physical and virtual models, and computing
the loss. On this account, accuracy is the main factor to
consider when evaluating digital twins. On the other hand,
the purpose of building a digital twin also matters in eval-
uating its success. This can be justified by the performance
improvement of the corresponding physical system that is
attributed to its digital twin. For example, for a DT whose
purpose is to optimize the assembly line, the improvement
can be measured by computing the number of actions (or sub-
tasks) and the time taken to manufacture a full component
(or to complete a main task/goal) with the DT and without
DT. This is also the case with other applications, includ-
ing product design optimization, product performance opti-
mization, process optimization, control optimization, sched-
uler optimization, resource management, component PHM,
etc. In addition, the processing time and efficiency of the
digital twinning system can also be one of the success
criteria.

In addition, when using AI or machine learning
approaches, the accuracy of the selected model affects the
efficacy of the DT. Specifically, the accuracy of the underly-
ing ML model, as well as the feature selection process and

the amount of training data, may greatly affect the outcome
of the DT. Therefore, when designing a DT-based system
that employs ML techniques, we have to select the model
with the higher accuracy and efficiency. The same approach
should be taken with the selection of other technologies for
DT-development, such as IoT, edge computing, and cloud
computing.

To this end, only a few state-of-the-art digital twinning sys-
tems have been fully evaluated in the literature. For instance,
Zhang et al. [87] assessed their job-floor digital twin by
comparing the performance of the job-floor with and without
digital twinning. They selected job scheduling time, utility
rate, and job tardiness as performance parameters. Similarly,
Zhang et al. [93] highlighted the importance of digital twin-
ning by showing the performance improvement in process
time, fault time, and maintenance time of blisk machining due
to its digital twin. Likewise, Min et al. [164] conveyed a rise
in the oil yield ratio due to a petrochemical industry DT. Fur-
thermore, Xu et al. [45] used the accuracy of fault diagnosis
as a metric to assess the performance of the developed virtual
twin. Finally, Akhlaghi et al. [99] verified the accuracy of the
developed twin by comparing the outputs of the digital and
physical twins. They also showed the effectiveness of their
digital twinning mechanism, by pointing out the optimization
achieved for the dew point cooler. All the aforementioned
DTs were developed using various machine learning models
and, in each case, the authors selected the model that provided
the best accuracy.

X. DIGITAL TWIN DEVELOPMENT TOOLS
There is no standalone technology for DT implementa-
tion, rather, there is an integration of multiple technologies,
including big data, AI-ML, IoT, CPS, edge computing, cloud
computing, communication technologies, etc. Every tech-
nological component can be implemented with a variety
of tools. Here, we only focus on the tools that facilitate
components integration, digital twin simulation, twins bridg-
ing, physical twin control, data storage and processing, and
machine learning. Table 4 presents the summary of widely
used tools that may provide support in different stages of
digital twinning.

Integrating physical components for data collection and
then digitally mimicking them in a virtual environment are
two important stages of digital twinning. There are various
tools available to accomplish these tasks in an industrial unit.
Siemens MindSphere is one of the widely used tools to inte-
grate components in a manufacturing industry. Siemens also
developed an object-oriented-based Tecnomatix API to sim-
ulate physical components in a virtual environment, as used
by [91]. The Open Simulation Platform (OSP) is another one,
which is jointly developed by the Det Norske Veritas Ger-
manischer Lloyd group (DNV GL), the Norwegian Univer-
sity of Science and Technology (NTNU), Rolls-Royce, and
SINTEF Ocean. OSP can digitally mimic any component of
the maritime industry. Other popular integration and simula-
tion tools are FIWARE, Predix (a cloud-based platform from

32044 VOLUME 9, 2021

M. M. Rathore et al.: Role of AI, Machine Learning, and Big Data in Digital Twinning: A SLR, Challenges, and Opportunities

TABLE 4. Digital twinning supporting tools.

VOLUME 9, 2021 32045

M. M. Rathore et al.: Role of AI, Machine Learning, and Big Data in Digital Twinning: A SLR, Challenges, and Opportunities

GE digital), CNC machine tools control platform IndraMo-
tion MTX, Beacon, Thingworx, and others.

Next, bridging physical and virtual twins is another pri-
mary aspect of digital twinning. This bridge is used by a vir-
tual twin to harvest the real-time data from the corresponding
physical peer using sensors. On the other side, the physical
peer is controlled (optimized) based on the output of the
virtual twin. Popular tools in the market to facilitate the bridg-
ing between physical and virtual twins are TwinCat, SAP,
Codesys, CNC tools, Aspera, and RaySync. Similarly, there
are few applications that are used in initial modeling and twin
design, such as ANSYS Twin Builder, MWorks, Siemens NX
software, SolidWorks, Autodesk tools, and FreeCAD.

In the machine learning domain, there are hundreds of
models available for tasks such as optimization, prediction,
classification, and clustering. However, there is no single
platform that offers APIs for all existing ML models. The
most widely used and well-known libraries for implement-
ing, training, and testing supervised ML-models are Tensor-
flow, CNTK and Caffe. Keras and Weka provide easier and
user-friendly interfaces for developing basic machine learn-
ing models. There are also commercial tools available, such
as Mathworks Matlab, which is equipped with vast libraries
of neural networks and Microsoft-Azure implemented ML
models. Reinforcement learning is one of the most popular
techniques that is widely used for dynamic optimization and
process planning in digital twinning. To this end, OpenAI’s
Gym and rllab are tools with standardized interfaces for
reinforcement learning.

Industrial components produce large amounts of data,
termed as big data, which are hard to process with standard
data management tools in a digital twin environment. Hadoop
is one of the most popular ecosystems for big data processing
that offers parallel processing capabilities with multiple com-
pute nodes. Apache has also developed several tools for big
data processing and effective analysis, including Cassandra,
Spark, Storm, S4, Hive, Mahout, Flink, and HBase. Most
of the Apache tools are open-source and support machine
learning APIs. Similar tools include HPCC by LexisNexis
Risk Solution, Qubole, Statwing, Pentaho, and VoltDB.

XI. DATA-DRIVEN REFERENCE ARCHITECTURE FOR
DIGITAL TWINNING
To effectively exploit the value-added capabilities offered by
the integration of big data analytics and AI-ML within the
scope of digital twinning, we present a novel reference model
derived from the conducted systematic literature review.
Fig. 8 shows the designed reference layered-architecture for
the efficient handling of big data analytics in DT-based indus-
trial environments. The process starts with the collection of
data from the physical environment (using sensors and actu-
ators) or from the virtual environment (using computer-aided
software and/or simulations). The data is fed to the data anal-
ysis and decision-making layer, where AI models, statistical
and probabilistic approaches, or mathematical models are
employed to create the DT-based system or the digital twin

itself. During the entire process, various big data processing
tools may be utilized, such as Hadoop, Storm, S4, Spark,
etc., that allow for parallel processing on multiple compute
nodes. Fig. 9 depicts the overall data flow for creating an
ML-enabled digital twin, and then using it for optimization,
PHM, or other purposes. First, the virtual model is created
by deploying one of the AI models on the data generated by
the physical twin. Once the digital twin is produced, the data
from both the physical and virtual twins are given to other
AI models to achieve the given industrial goals, such as
design optimization, dynamic process planning, healthcare,
or PHM. Moreover, the results can be further used to update
and improve both the physical and virtual twins.

XII. MARKET OPPORTUNITIES AND RESEARCH
CHALLENGES
A. MARKET OPPORTUNITIES AND RESEARCH AREAS
Based on the detailed literature survey, we have summarized
the following major application areas where DT research can
play a vital role.

1) OPTIMIZATION
Optimization is required in almost every industrial
process, including product design, product performance, pro-
cess planning, assembly line, task-scheduling, and resource-
allocation. Digital twinning is an emerging technology that
provides a direct pathway to optimization with little effort.
However, careful consideration of the optimization algorithm
(i.e., ML model) and the underlying feature set (for the
optimization algorithm) is desired for better results.

2) PROCESS MONITORING, DIAGNOSTICS, AND
PREDICTION
Digital twins can be developed for industrial process mon-
itoring, defect diagnosis (i.e., product quality assurance),
dynamic process or product design updating for time and
cost savings, industrial process surveillance (e.g., robot DT
for obstacle avoidance), product time-to-complete prediction,
and damage detection.

3) PREDICTIVE ANALYTICS FOR MANUFACTURED
PRODUCTS
The quality of every physical entity degrades over time, thus
affecting its performance. Early detection of failures may
promote on-time maintenance, fatigue avoidance, as well as
time and cost savings. Such failures can be attributed to faults
and cracks in the product, performance degradation due to
aging, and other minor or major complications. Moreover,
health monitoring is crucial for certain components that may
potentially cause human casualties, e.g., brake systems in
cars, vehicles, aircraft, and ship engines, fueling systems,
gearboxes, etc. Digital twinning is the most powerful technol-
ogy for predictive analytics and health monitoring of physical
components. This is also an area where AI-ML techniques
can have a significant impact.

32046 VOLUME 9, 2021

M. M. Rathore et al.: Role of AI, Machine Learning, and Big Data in Digital Twinning: A SLR, Challenges, and Opportunities

FIGURE 8. Data-driven reference architecture for digital twinning.

FIGURE 9. Overall data-flow framework for digital twinning using big
data analytics and AI-ML.

4) HEALTHCARE
Digital twinning has a wider scope in the healthcare sector
where human-DTs assist in day-to-day human fitness and
health monitoring, early disease diagnosis, and the over-
all well-being of individuals, especially for the elderly and
infants. In addition, it can be used for the treatment or
surgery of patients, by developing a patient-DT. Developing
digital twins for human organs or biological systems will
bring a revolution in the healthcare sector, such as DTs for
lungs, liver, pregnant female womb or uterus, cardiac system,
digestion system, neural system, reproductive system, etc.
Other than biological digital twins, the healthcare sector can
benefit by developing DTs for hospitals, medical and surgical
instruments, remote surgery, surgical processes, etc.

5) SMART CITIES
In the context of smart cities, DT technologies can be imple-
mented for traffic systems, smart homes and devices, park-
ing, buildings, livestock, lighting systems, and renewable
energy. Furthermore, 3D virtual city models may facilitate
urban planning and monitoring in various smart city areas,
including road monitoring and construction, city garbage
management, bridge and housing constructions, etc.

6) OTHER APPLICATIONS
Research opportunities are not limited to the above-mentioned
sectors, but the potential is there in every field, including edu-
cation, construction, mining, communications and networks,
food and agriculture, sports, and so on.

B. RESEARCH CHALLENGES AND ISSUES
The rapidly increasing DT popularity and scope, as well as the
involvement of IoT, big data, and AI technologies, broaden
the research challenges of digital twinning. These challenges
are categorized in the following five areas.

1) DATA COLLECTION
IoT facilitates data harvesting from a physical twin (using
sensors), data integration, and data sharing with the corre-
sponding virtual twins. This process can amount to a consid-
erable cost. Sometimes, the digital twin may be more costly
than the asset itself, in which case it does not make sense to
create the DT. On the other hand, the collected data is large
(big data), heterogeneous in nature, unstructured, and noisy.
Thus, further processing on the data is required to ensure
its effective use. Specifically, we need to apply data clean-
ing techniques, and also organize, restructure, and make the
data homogeneous. Furthermore, controlling the flow of such
large amount of data is also a significant challenge. Finally,
to improve the accuracy of the DT model, the underlying

VOLUME 9, 2021 32047

M. M. Rathore et al.: Role of AI, Machine Learning, and Big Data in Digital Twinning: A SLR, Challenges, and Opportunities

machine learning algorithms require a certain amount of data
for training purposes.

2) BIG DATA CHALLENGES
The explosive growth of IoT technologies in the industrial
sector has led to the generation of large amounts of moni-
toring (sensor) data. To this end, big data analytics requires
advanced architectures, frameworks, technologies, tools, and
algorithms to capture, store, share, process, and analyze the
underlying data. There is also a potential for edge and cloud
computing platforms to handle DT-related data. Specifically,
edge computing enables the distributed processing at the net-
work’s edge, while the aggregate processing is accomplished
in the cloud. However, the aggregation of data in the cloud
may cause an increase in response time.

3) DATA ANALYSIS
AI-algorithms for data analytics played a major role in DT
for decision-making, as discussed in the literature. How-
ever, the selection of a particular model among hundreds of
ML-models with customized configuration is challenging.
Every AI-approach has diverse accuracy and efficiency levels
with different applications and datasets (feature set). On the
other hand, accuracy may affect the efficiency on the other
side. Hence, depending on the motive and application of
a DT, the selection of the best ML-algorithm and features
is challenging. Besides, fewer practical implementations of
AI-techniques for digital twinning in the literature raises
more challenges.

4) DT STANDARDIZATION CHALLENGES
Even though many digital twins have been developed in var-
ious industries, the creation of a complex and reliable digital
twin demands standardization. Currently, there is no single
standard that solely focuses on digital twinning. The ISO/DIS
23247-1 standard [29] has only limited information on digital
twinning and, therefore, DT deployment challenges grow due
to the lack of standardization. Standardization efforts are
underway by the joint advisory group (JAG) of ISO and IEC
on emerging technologies [28].

5) SECURITY AND PRIVACY ISSUES
Some DT systems, such as human-DTs, product PHM,
or defense-related DTs, are considered critical and may
require stringent security and privacy guarantees. First, due
to the involvement of IoT devices in digital twinning, a lot
of emphasis has to be placed on the security of the under-
lying communication protocols. Additionally, the large col-
lection of asset-related data needs to be stored securely,
in order to prevent data breaches from insider and outsider
threats.

XIII. CONCLUSION
We performed a systematic literature review of the state-of-
the-art DT systems that employ machine learning and AI
technologies. In particular, we focused on papers published

in top multidisciplinary electronic bibliographic and patent
libraries, and summarized the current DT deployments in
a variety of industries. With the immersion of AI-ML and
big data, digital twinning is evolving at a rapid rate and,
with it, a lot of unique challenges and new opportunities are
emerging. This article highlighted the research challenges
and potentials in many diverse areas, for both academia and
industry. Furthermore, we identified the DT criteria and tools
that aid its successful development. Finally, we designed a
reference model for an AI-ML and big data-enabled digital
twinning system to further guide industrial developers in
establishing DTs that can make their systems smarter, intelli-
gent, and dynamically adaptable to changing conditions.

REFERENCES
[1] M. W. Grieves, ‘‘Virtually intelligent product systems: Digital and

physical twins,’’ Complex Syst. Eng., Theory Pract., pp. 175–200,
2019.

[2] M. Grieves, ‘‘Digital twin: Manufacturing excellence through virtual
factory replication,’’ White Paper, 2014, pp. 1–7, vol. 1.

[3] E. J. Tuegel, A. R. Ingraffea, T. G. Eason, and S. M. Spottswood,
‘‘Reengineering aircraft structural life prediction using a digital twin,’’
Int. J. Aerosp. Eng., vol. 2011, pp. 1–14, Aug. 2011.

[4] D. Cearley, B. Burke, D. Smith, N. Jones, A. Chandrasekaran, and C. Lu,
‘‘Top 10 strategic technology trends for 2020,’’ Gartner, Stamford, CT,
USA, Tech. Rep., 2019.

[5] T. R. Wanasinghe, L. Wroblewski, B. K. Petersen, R. G. Gosine,
L. A. James, O. De Silva, G. K. I. Mann, and P. J. Warrian, ‘‘Digital twin
for the oil and gas industry: Overview, research trends, opportunities, and
challenges,’’ IEEE Access, vol. 8, pp. 104175–104197, 2020.

[6] Y. Lu, C. Liu, K. I.-K. Wang, H. Huang, and X. Xu, ‘‘Digital twin-
driven smart manufacturing: Connotation, reference model, applications
and research issues,’’ Robot. Comput.-Integr. Manuf., vol. 61, Feb. 2020,
Art. no. 101837.

[7] C. Cimino, E. Negri, and L. Fumagalli, ‘‘Review of digital twin
applications in manufacturing,’’ Comput. Ind., vol. 113, Dec. 2019,
Art. no. 103130.

[8] Q. Qi and F. Tao, ‘‘Digital twin and big data towards smart manufac-
turing and industry 4.0: 360 degree comparison,’’ IEEE Access, vol. 6,
pp. 3585–3593, 2018.

[9] F. Tao, H. Zhang, A. Liu, and A. Y. C. Nee, ‘‘Digital twin in
industry: State-of-the-art,’’ IEEE Trans. Ind. Informat., vol. 15, no. 4,
pp. 2405–2415, Apr. 2019.

[10] A. Rasheed, O. San, and T. Kvamsdal, ‘‘Digital twin: Values, chal-
lenges and enablers from a modeling perspective,’’ IEEE Access, vol. 8,
pp. 21980–22012, 2020.

[11] B. Kitchenham and S. Charters, ‘‘Guidelines for performing systematic
literature reviews in software engineering,’’ Keele Univ., Durham Univ.,
Keele, U.K., Tech. Rep. EBSE 2007-001, 2007.

[12] B. Kitchenham, O. P. Brereton, D. Budgen, M. Turner, J. Bailey, and
S. Linkman, ‘‘Systematic literature reviews in software engineering—
A systematic literature review,’’ Inf. Softw. Technol., vol. 51, no. 1,
pp. 7–15, Jan. 2009.

[13] C. Okoli and K. Schabram, ‘‘A guide to conducting a systematic litera-
ture review of information systems research,’’ SSRN, Tech. Rep., 2010.
[Online]. Available: http://dx.doi.org/10.2139/ssrn.1954824

[14] D. Cearley, B. Burke, S. Searle, and M. Walker, ‘‘Top 10 strategic tech-
nology trends for 2017: A gartner trend insight report,’’ Gartner, vol. 23,
Jun. 2017, Art. no. 6595640781. [Online]. Available: https://www.
gartner.com/doc/3645332

[15] D. Cearley, B. Burke, S. Searle, and M. J. Walker, ‘‘Top 10 strategic
technology trends for 2018,’’ Gartner, 2017.

[16] D. Cearley and B. Burke, ‘‘Top 10 strategic technology trends for 2019,’’
Gartner, 2018.

[17] M. Grieves and J. Vickers, ‘‘Digital twin: Mitigating unpredictable, unde-
sirable emergent behavior in complex systems,’’ in Transdisciplinary
Perspectives on Complex Systems. Cham, Switzerland: Springer, 2017,
pp. 85–113.

32048 VOLUME 9, 2021

M. M. Rathore et al.: Role of AI, Machine Learning, and Big Data in Digital Twinning: A SLR, Challenges, and Opportunities

[18] E. Glaessgen and D. Stargel, ‘‘The digital twin paradigm for future NASA
and US air force vehicles,’’ in Proc. 53rd AIAA/ASME/ASCE/AHS/ASC
Struct., Struct. Dyn. Mater. Conf., 20th AIAA/ASME/AHS Adapt. Struct.
Conf., 14th AIAA, 2012, p. 1818.

[19] F. Tao, F. Sui, A. Liu, Q. Qi, M. Zhang, B. Song, Z. Guo, S. C.-Y. Lu, and
A. Nee, ‘‘Digital twin-driven product design framework,’’ Int. J. Prod.
Res., vol. 57, no. 12, pp. 3935–3953, 2019.

[20] R. Söderberg, K. Wärmefjord, J. S. Carlson, and L. Lindkvist, ‘‘Toward
a digital twin for real-time geometry assurance in individualized produc-
tion,’’ CIRP Ann., vol. 66, no. 1, pp. 137–140, 2017.

[21] G. Bacchiega, ‘‘Creating an embedded digital twin: Monitor, understand
and predict device health failure,’’ Inn4mech-Mechatronics Ind., vol. 4,
2018.

[22] R. Dong, C. She, W. Hardjawana, Y. Li, and B. Vucetic, ‘‘Deep learn-
ing for hybrid 5G services in mobile edge computing systems: Learn
from a digital twin,’’ IEEE Trans. Wireless Commun., vol. 18, no. 10,
pp. 4692–4707, Oct. 2019.

[23] B. Björnsson, C. Borrebaeck, N. Elander, T. Gasslander,
D. R. Gawel, M. Gustafsson, R. Jörnsten, E. J. Lee, X. Li, S. Lilja,
D. Martínez-Enguita, A. Matussek, P. Sandström, S. Schäfer,
M. Stenmarker, X. F. Sun, O. Sysoev, H. Zhang, and M. Benson,
‘‘Digital twins to personalize medicine,’’ Genome Med., vol. 12, no. 1,
pp. 1–4, Dec. 2020.

[24] J. A. Marmolejo-Saucedo, ‘‘Design and development of digital twins:
A case study in supply chains,’’ Mobile Netw. Appl., vol. 25, no. 6,
pp. 2141–2160, Dec. 2020.

[25] C. Zhuang, J. Liu, and H. Xiong, ‘‘Digital twin-based smart pro-
duction management and control framework for the complex product
assembly shop-floor,’’ Int. J. Adv. Manuf. Technol., vol. 96, nos. 1–4,
pp. 1149–1163, Apr. 2018.

[26] R. Piascik, J. Vickers, D. Lowry, S. Scotti, J. Stewart, and A. Calomino,
‘‘Technology area 12: Materials, structures, mechanical systems, and
manufacturing road map,’’ NASA Office Chief Technol., 2010.

[27] P. Caruso, D. Dumbacher, and M. Grieves, ‘‘Product lifecycle manage-
ment and the quest for sustainable space exploration,’’ in Proc. AIAA
SPACE Conf. Expo., Aug. 2010, p. 8628.

[28] JETI. Which Technologies is Jeti Considering? Accessed: May 8, 2020.
[Online]. Available: https://jtc1info.org/technology/advisory-groups/jeti/

[29] Automation Systems and Integration Digital Twin Framework for
Manufacturing—Part 1: Overview and General Principles, Stan-
dard ISO/DIS 23247-1, 2020. [Online]. Available: https://www.iso.
org/standard/75066.html

[30] Industrial Automation Systems and Integration-Product Data Repre-
sentation and Exchange—Part 1: Overview and Fundamental Princi-
ples, Standard ISO 10303-1, 1994. [Online]. Available: https://www.iso.
org/standard/20579.html

[31] 2014CuttingToolDataRepresentationandExchange—Part3:Reference
Dictionary for Tool Items, Int. Org. Standard, Standard ISO 13399-3,
2014. [Online]. Available: https://www.iso.org/standard/54168.html

[32] O. Foundation. Unified Architecture. Accessed: 2008. [Online]. Avail-
able: https://opcfoundation.org/about/opc-technologies/opc-ua/

[33] R. Rosen, G. von Wichert, G. Lo, and K. D. Bettenhausen, ‘‘About the
importance of autonomy and digital twins for the future of manufactur-
ing,’’ IFAC-PapersOnLine, vol. 48, no. 3, pp. 567–572, 2015.

[34] J. Vachálek, L. Bartalský, O. Rovný, D. Šišmišová, M. Morhác, and
M. Lokšík, ‘‘The digital twin of an industrial production line within the
industry 4.0 concept,’’ in Proc. 21st Int. Conf. Process Control (PC),
Jun. 2017, pp. 258–262.

[35] E. Negri, S. Berardi, L. Fumagalli, and M. Macchi, ‘‘MES-integrated
digital twin frameworks,’’ J. Manuf. Syst., vol. 56, pp. 58–71, Jul. 2020.

[36] Z. Yin and J. Hou, ‘‘Recent advances on SVM based fault diagnosis and
process monitoring in complicated industrial processes,’’ Neurocomput-
ing, vol. 174, pp. 643–650, Jan. 2016.

[37] L. Bennacer, Y. Amirat, A. Chibani, A. Mellouk, and L. Ciavaglia, ‘‘Self-
diagnosis technique for virtual private networks combining Bayesian net-
works and case-based reasoning,’’ IEEE Trans. Autom. Sci. Eng., vol. 12,
no. 1, pp. 354–366, Jan. 2015.

[38] P. Tamilselvan and P. Wang, ‘‘Failure diagnosis using deep belief learn-
ing based health state classification,’’ Rel. Eng. Syst. Saf., vol. 115,
pp. 124–135, Jul. 2013.

[39] Y. Qi, C. Shen, D. Wang, J. Shi, X. Jiang, and Z. Zhu, ‘‘Stacked sparse
autoencoder-based deep network for fault diagnosis of rotating machin-
ery,’’ IEEE Access, vol. 5, pp. 15066–15079, 2017.

[40] W. Lu, Y. Li, Y. Cheng, D. Meng, B. Liang, and P. Zhou, ‘‘Early fault
detection approach with deep architectures,’’ IEEETrans. Instrum.Meas.,
vol. 67, no. 7, pp. 1679–1689, Jul. 2018.

[41] Y. Qi Chen, O. Fink, and G. Sansavini, ‘‘Combined fault location and
classification for power transmission lines fault diagnosis with inte-
grated feature extraction,’’ IEEE Trans. Ind. Electron., vol. 65, no. 1,
pp. 561–569, Jan. 2018.

[42] H. Darong, K. Lanyan, M. Bo, Z. Ling, and S. Guoxi, ‘‘A new incipient
fault diagnosis method combining improved RLS and LMD algorithm
for rolling bearings with strong background noise,’’ IEEE Access, vol. 6,
pp. 26001–26010, 2018.

[43] Y. Wang, Z. Wei, and J. Yang, ‘‘Feature trend extraction and adaptive
density peaks search for intelligent fault diagnosis of machines,’’ IEEE
Trans. Ind. Informat., vol. 15, no. 1, pp. 105–115, Jan. 2019.

[44] S. Yin, X. Zhu, and O. Kaynak, ‘‘Improved PLS focused on key-
performance-indicator-related fault diagnosis,’’ IEEE Trans. Ind. Elec-
tron., vol. 62, no. 3, pp. 1651–1658, Mar. 2015.

[45] Y. Xu, Y. Sun, X. Liu, and Y. Zheng, ‘‘A digital-twin-assisted fault diagno-
sis using deep transfer learning,’’ IEEE Access, vol. 7, pp. 19990–19999,
2019.

[46] Y. Wang, R. Xiong, H. Yu, J. Zhang, and Y. Liu, ‘‘Perception of demon-
stration for automatic programing of robotic assembly: Framework, algo-
rithm, and validation,’’ IEEE/ASME Trans. Mechatronics, vol. 23, no. 3,
pp. 1059–1070, Jun. 2018.

[47] X. Li, B. He, Y. Zhou, and G. Li, ‘‘Multisource model-driven digital twin
system of robotic assembly,’’ IEEE Syst. J., early access, Jan. 3, 2020,
doi: 10.1109/JSYST.2019.2958874.

[48] Y. Fang, C. Peng, P. Lou, Z. Zhou, J. Hu, and J. Yan, ‘‘Digital-twin-
based job shop scheduling toward smart manufacturing,’’ IEEE Trans.
Ind. Informat., vol. 15, no. 12, pp. 6425–6435, Dec. 2019.

[49] S. Scharff. (2019). From Digital Twin to Improved Patient Experi-
ence. Accessed: May 8, 2020. [Online]. Available: https://www.siemens-
healthineers.com/news/mso-digital-twin-mater.html

[50] T. Marchal. (Sep. 2016). VPH: The Ultimate Stage Before Your Own
Medical Digital Twin. Accessed: May 8, 2020. [Online]. Available:
https://www.linkedin.com/pulse/vph-ultimate-stage-before-your-own-
medical-digital-twin-marchal/?trk=mp-reader-car

[51] C. Copley. (Aug. 2018). Medical Technology Firms Develop ‘Dig-
ital Twins’ for Personalized Health Care. Accessed: May 8, 2020.
[Online]. Available: https://www.theglobeandmail.com/business/article-
medical-technology-firms-develop-digital-twins-for-personalized/

[52] R. Martinez-Velazquez, R. Gamez, and A. El Saddik, ‘‘Cardio twin:
A digital twin of the human heart running on the edge,’’ in Proc. IEEE
Int. Symp. Med. Meas. Appl. (MeMeA), Jun. 2019, pp. 1–6.

[53] J. M. Ospel, G. Gascou, V. Costalat, L. Piergallini, K. A. Blackham,
and D. W. Zumofen, ‘‘Comparison of Pipeline embolization device sizing
based on conventional 2D measurements and virtual simulation using the
Sim&Size software: An agreement study,’’ Amer. J. Neuroradiol., vol. 40,
no. 3, pp. 524–530, Feb. 2019.

[54] M. Holtmannspotter, M. Martinez-Galdamez, M. Isokangas, R. Ferrara,
and M. Sanchez, ‘‘Simulation in clinical practice: First experience with
Sim&Cure before implantation of flow diverter (pipeline) or web-device
for the treatment of intracranial aneurysm,’’ in Proc. ABC/WIN, 2017.

[55] Y. Feng, J. Zhao, X. Chen, and J. Lin, ‘‘An in silico subject-variability
study of upper airway morphological influence on the airflow regime in
a tracheobronchial tree,’’ Bioengineering, vol. 4, no. 4, p. 90, Nov. 2017.

[56] Y. Liu, L. Zhang, Y. Yang, L. Zhou, L. Ren, F. Wang, R. Liu, Z. Pang, and
M. J. Deen, ‘‘A novel cloud-based framework for the elderly healthcare
services using digital twin,’’ IEEEAccess, vol. 7, pp. 49088–49101, 2019.

[57] Z. Wang, X. Liao, X. Zhao, K. Han, P. Tiwari, M. J. Barth, and G. Wu,
‘‘A digital twin paradigm: Vehicle-to-cloud based advanced driver assis-
tance systems,’’ in Proc. IEEE 91st Veh. Technol. Conf. (VTC-Spring),
May 2020, pp. 1–6.

[58] E. Cioroaica, T. Kuhn, and B. Buhnova, ‘‘(Do Not) trust in ecosystems,’’
in Proc. IEEE/ACM 41st Int. Conf. Softw. Eng., New Ideas Emerg. Results
(ICSE-NIER), May 2019, pp. 9–12.

[59] X. Chen, T. Chen, Z. Zhao, H. Zhang, M. Bennis, and J. I. Yusheng,
‘‘Resource awareness in unmanned aerial vehicle-assisted mobile-edge
computing systems,’’ in Proc. IEEE 91st Veh. Technol. Conf. (VTC-
Spring), May 2020, pp. 1–6.

[60] S. A. P. Kumar, R. Madhumathi, P. R. Chelliah, L. Tao, and S. Wang,
‘‘A novel digital twin-centric approach for driver intention prediction and
traffic congestion avoidance,’’ J. Reliable Intell. Environ., vol. 4, no. 4,
pp. 199–209, Dec. 2018.

VOLUME 9, 2021 32049

http://dx.doi.org/10.1109/JSYST.2019.2958874

M. M. Rathore et al.: Role of AI, Machine Learning, and Big Data in Digital Twinning: A SLR, Challenges, and Opportunities

[61] S. Venkatesan, K. Manickavasagam, N. Tengenkai, and
N. Vijayalakshmi, ‘‘Health monitoring and prognosis of electric
vehicle motor using intelligent-digital twin,’’ IET Electr. Power Appl.,
vol. 13, no. 9, pp. 1328–1335, Sep. 2019.

[62] S. M. E. Sepasgozar, ‘‘Digital twin and Web-based virtual gaming tech-
nologies for online education: A case of construction management and
engineering,’’ Appl. Sci., vol. 10, no. 13, p. 4678, Jul. 2020.

[63] M. Lammers. (Jun. 2018). Opinion | Digital Twin Offers Huge Oppor-
tunities for Real Estate Life Cycle. Accessed: May 8, 2020. [Online].
Available: https://www.proptech.nl/blog/digital-twin/

[64] A. Kampker, V. Stich, P. Jussen, B. Moser, and J. Kuntz, ‘‘Business
models for industrial smart services—The example of a digital twin for
a product-service-system for potato harvesting,’’ Procedia CIRP, vol. 83,
pp. 534–540, Jan. 2019.

[65] S. H. Khajavi, N. H. Motlagh, A. Jaribion, L. C. Werner, and
J. Holmström, ‘‘Digital twin: Vision, benefits, boundaries, and creation
for buildings,’’ IEEE Access, vol. 7, pp. 147406–147419, 2019.

[66] C. Fan, C. Zhang, A. Yahja, and A. Mostafavi, ‘‘Disaster city digital
twin: A vision for integrating artificial and human intelligence for disaster
management,’’ Int. J. Inf. Manage., vol. 56, Feb. 2021, Art. no. 102049.

[67] M. Dietz and G. Pernul, ‘‘Unleashing the digital Twin’s potential for ICS
security,’’ IEEE Secur. Privacy, vol. 18, no. 4, pp. 20–27, Jul. 2020.

[68] R. Langner, ‘‘To kill a centrifuge: A technical analysis of what stuxnet’s
creators tried to achieve,’’ The Langner Group, Tech. Rep., 2013.

[69] S. Miller, N. Brubaker, D. K. Zafra, and D. Caban, ‘‘Triton actor TTP
profile, custom attack tools, detections, and ATT&CK mapping,’’ Fireeye
Threat Res. Blog, Apr. 2019.

[70] M. M. U. Rathore, M. J. J. Gul, A. Paul, A. A. Khan, R. W. Ahmad,
J. Rodrigues, and S. Bakiras, ‘‘Multilevel graph-based decision mak-
ing in big scholarly data: An approach to identify expert reviewer,
finding quality impact factor, ranking journals and researchers,’’ IEEE
Trans. Emerg. Topics Comput., early access, Sep. 10, 2018, doi:
10.1109/TETC.2018.2869458.

[71] M. M. Rathore, H. Son, A. Ahmad, and A. Paul, ‘‘Real-time video
processing for traffic control in smart city using Hadoop ecosystem with
GPUs,’’ Soft Comput., vol. 22, no. 5, pp. 1533–1544, Mar. 2018.

[72] M. M. Rathore, A. Ahmad, A. Paul, and S. Rho, ‘‘Exploiting encrypted
and tunneled multimedia calls in high-speed big data environment,’’
Multimedia Tools Appl., vol. 77, no. 4, pp. 4959–4984, Feb. 2018.

[73] S. A. Shah, D. Z. Seker, M. M. Rathore, S. Hameed, S. Ben Yahia,
and D. Draheim, ‘‘Towards disaster resilient smart cities: Can Internet
of Things and big data analytics be the game changers?’’ IEEE Access,
vol. 7, pp. 91885–91903, 2019.

[74] X. Yuan, C. J. Anumba, and M. K. Parfitt, ‘‘Cyber-physical systems for
temporary structure monitoring,’’ Autom. Construct., vol. 66, pp. 1–14,
Jun. 2016.

[75] F. Thiesse, M. Dierkes, and E. Fleisch, ‘‘LotTrack: RFID-based process
control in the semiconductor industry,’’ IEEE Pervas. Comput., vol. 5,
no. 1, pp. 47–53, Jan. 2006.

[76] H. Choi, Y. Baek, and B. Lee, ‘‘Design and implementation of practical
asset tracking system in container terminals,’’ Int. J. Precis. Eng. Manuf.,
vol. 13, no. 11, pp. 1955–1964, Nov. 2012.

[77] Y. Zheng, S. Yang, and H. Cheng, ‘‘An application framework of digital
twin and its case study,’’ J. Ambient Intell. Humanized Comput., vol. 10,
no. 3, pp. 1141–1153, Mar. 2019.

[78] K. Ding, H. Shi, J. Hui, Y. Liu, B. Zhu, F. Zhang, and W. Cao, ‘‘Smart steel
bridge construction enabled by BIM and Internet of Things in industry
4.0: A framework,’’ in Proc. IEEE 15th Int. Conf. Netw., Sens. Control
(ICNSC), Mar. 2018, pp. 1–5.

[79] Y. Zhou, T. Xing, Y. Song, Y. Li, X. Zhu, G. Li, and S. Ding, ‘‘Digital-
twin-driven geometric optimization of centrifugal impeller with free-form
blades for five-axis flank milling,’’ J. Manuf. Syst., Jul. 2020.

[80] A. Oluwasegun and J.-C. Jung, ‘‘The application of machine learning for
the prognostics and health management of control element drive system,’’
Nucl. Eng. Technol., vol. 52, no. 10, pp. 2262–2273, Oct. 2020.

[81] A. Gandomi and M. Haider, ‘‘Beyond the hype: Big data concepts,
methods, and analytics,’’ Int. J. Inf. Manage., vol. 35, no. 2, pp. 137–144,
Apr. 2015.

[82] Y. Zhang, S. Ma, H. Yang, J. Lv, and Y. Liu, ‘‘A big data driven analytical
framework for energy-intensive manufacturing industries,’’ J. Cleaner
Prod., vol. 197, pp. 57–72, Oct. 2018.

[83] Y. Zhang, S. Ren, Y. Liu, and S. Si, ‘‘A big data analytics architecture for
cleaner manufacturing and maintenance processes of complex products,’’
J. Cleaner Prod., vol. 142, pp. 626–641, Jan. 2017.

[84] Y. Wang, S. Wang, B. Yang, L. Zhu, and F. Liu, ‘‘Big data driven hier-
archical digital twin predictive remanufacturing paradigm: Architecture,
control mechanism, application scenario and benefits,’’ J. Cleaner Prod.,
vol. 248, Mar. 2020, Art. no. 119299.

[85] K. M. Alam and A. El Saddik, ‘‘C2PS: A digital twin architecture refer-
ence model for the cloud-based cyber-physical systems,’’ IEEE Access,
vol. 5, pp. 2050–2062, 2017.

[86] E. A. Patterson, R. J. Taylor, and M. Bankhead, ‘‘A framework for an
integrated nuclear digital environment,’’ Prog. Nucl. Energy, vol. 87,
pp. 97–103, Mar. 2016.

[87] M. Zhang, F. Tao, and A. Y. C. Nee, ‘‘Digital twin enhanced dynamic
job-shop scheduling,’’ J. Manuf. Syst., May 2020.

[88] M. Schluse, M. Priggemeyer, L. Atorf, and J. Rossmann, ‘‘Experi-
mentable digital twins—Streamlining simulation-based systems engi-
neering for industry 4.0,’’ IEEE Trans. Ind. Informat., vol. 14, no. 4,
pp. 1722–1731, Feb. 2018.

[89] S. Zhang, C. Kang, Z. Liu, J. Wu, and C. Ma, ‘‘A product quality monitor
model with the digital twin model and the stacked auto encoder,’’ IEEE
Access, vol. 8, pp. 113826–113836, 2020.

[90] R. Bansal, M. A. Khanesar, and D. Branson, ‘‘Ant colony optimization
algorithm for industrial robot programming in a digital twin,’’ in Proc.
25th Int. Conf. Autom. Comput. (ICAC), Sep. 2019, pp. 1–5.

[91] K. Xia, C. Sacco, M. Kirkpatrick, C. Saidy, L. Nguyen, A. Kircaliali, and
R. Harik, ‘‘A digital twin to train deep reinforcement learning agent for
smart manufacturing plants: Environment, interfaces and intelligence,’’
J. Manuf. Syst., Jul. 2020.

[92] F. Tao, J. Cheng, Q. Qi, M. Zhang, H. Zhang, and F. Sui, ‘‘Digital twin-
driven product design, manufacturing and service with big data,’’ Int. J.
Adv. Manuf. Technol., vol. 94, nos. 9–12, pp. 3563–3576, Feb. 2018.

[93] H. Zhang, G. Zhang, and Q. Yan, ‘‘Digital twin-driven cyber-physical pro-
duction system towards smart shop-floor,’’ J. Ambient Intell. Humanized
Comput., vol. 10, no. 11, pp. 4439–4453, Nov. 2019.

[94] W. Wang, Y. Zhang, and R. Y. Zhong, ‘‘A proactive material handling
method for CPS enabled shop-floor,’’ Robot. Comput.-Integr. Manuf.,
vol. 61, Feb. 2020, Art. no. 101849.

[95] C. Zhang, G. Zhou, J. Hu, and J. Li, ‘‘Deep learning-enabled intelligent
process planning for digital twin manufacturing cell,’’ Knowl.-Based
Syst., vol. 191, Mar. 2020, Art. no. 105247.

[96] S. Liu, J. Bao, Y. Lu, J. Li, S. Lu, and X. Sun, ‘‘Digital twin modeling
method based on biomimicry for machining aerospace components,’’
J. Manuf. Syst., May 2020.

[97] J. Liu, H. Zhou, G. Tian, X. Liu, and X. Jing, ‘‘Digital twin-based process
reuse and evaluation approach for smart process planning,’’ Int. J. Adv.
Manuf. Technol., vol. 100, nos. 5–8, pp. 1619–1634, Feb. 2019.

[98] P. Franciosa, M. Sokolov, S. Sinha, T. Sun, and D. Ceglarek, ‘‘Deep
learning enhanced digital twin for remote laser welding of aluminium
structures,’’ CIRP Ann. Manuf. Technol., vol. 69, no. 1, 2020.

[99] Y. Golizadeh Akhlaghi, A. Badiei, X. Zhao, K. Aslansefat, X. Xiao,
S. Shittu, and X. Ma, ‘‘A constraint multi-objective evolutionary opti-
mization of a state-of-the-art dew point cooler using digital twins,’’
Energy Convers. Manage., vol. 211, May 2020, Art. no. 112772.

[100] C. Liu, L. Le Roux, C. Körner, O. Tabaste, F. Lacan, and S. Bigot,
‘‘Digital twin-enabled collaborative data management for metal additive
manufacturing systems,’’ J. Manuf. Syst., May 2020.

[101] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou,
D. Wierstra, and M. Riedmiller, ‘‘Playing atari with deep reinforcement
learning,’’ 2013, arXiv:1312.5602. [Online]. Available: http://arxiv.org/
abs/1312.5602

[102] H. Van Hasselt, A. Guez, and D. Silver, ‘‘Deep reinforcement learning
with double q-learning,’’ in Proc. 13th AAAI Conf. Artif. Intell., 2016,
pp. 1–7.

[103] T. Schaul, J. Quan, I. Antonoglou, and D. Silver, ‘‘Prioritized experience
replay,’’ 2015, arXiv:1511.05952. [Online]. Available: http://arxiv.org/
abs/1511.05952

[104] J. Leng, C. Jin, A. Vogl, and H. Liu, ‘‘Deep reinforcement learning
for a color-batching resequencing problem,’’ J. Manuf. Syst., vol. 56,
pp. 175–187, Jul. 2020.

[105] K. He, X. Zhang, S. Ren, and J. Sun, ‘‘Identity mappings in deep residual
networks,’’ in Proc.Eur.Conf.Comput.Vis. Cham, Switzerland: Springer,
2016, pp. 630–645.

[106] J. Lee, M. Azamfar, J. Singh, and S. Siahpour, ‘‘Integration of digital
twin and deep learning in cyber-physical systems: Towards smart man-
ufacturing,’’ IET Collaborative Intell. Manuf., vol. 2, no. 1, pp. 34–36,
Mar. 2020.

32050 VOLUME 9, 2021

http://dx.doi.org/10.1109/TETC.2018.2869458

M. M. Rathore et al.: Role of AI, Machine Learning, and Big Data in Digital Twinning: A SLR, Challenges, and Opportunities

[107] M. Tomko and S. Winter, ‘‘Beyond digital twins—A commentary,’’ Env-
iron. Planning B, Urban Analytics City Sci., vol. 46, no. 2, pp. 395–399,
Feb. 2019.

[108] F. Tao, M. Zhang, Y. Liu, and A. Y. C. Nee, ‘‘Digital twin driven prognos-
tics and health management for complex equipment,’’ CIRP Ann., vol. 67,
no. 1, pp. 169–172, 2018.

[109] G.-B. Huang, H. Zhou, X. Ding, and R. Zhang, ‘‘Extreme learning
machine for regression and multiclass classification,’’ IEEE Trans. Syst.,
Man, Cybern. B, Cybern., vol. 42, no. 2, pp. 513–529, Apr. 2012.

[110] A. Coraddu, L. Oneto, F. Baldi, F. Cipollini, M. Atlar, and S. Savio,
‘‘Data-driven ship digital twin for estimating the speed loss caused by
the marine fouling,’’ Ocean Eng., vol. 186, Aug. 2019, Art. no. 106063.

[111] J. Tang, C. Deng, and G.-B. Huang, ‘‘Extreme learning machine for
multilayer perceptron,’’ IEEE Trans. Neural Netw. Learn. Syst., vol. 27,
no. 4, pp. 809–821, Apr. 2016.

[112] P. Jain, J. Poon, J. P. Singh, C. Spanos, S. R. Sanders, and S. K. Panda,
‘‘A digital twin approach for fault diagnosis in distributed photovoltaic
systems,’’ IEEE Trans. Power Electron., vol. 35, no. 1, pp. 940–956,
Jan. 2020.

[113] W. Li, M. Rentemeister, J. Badeda, D. Jöst, D. Schulte, and D. U. Sauer,
‘‘Digital twin for battery systems: Cloud battery management system with
online state-of-charge and state-of-health estimation,’’ J. Energy Storage,
vol. 30, Aug. 2020, Art. no. 101557.

[114] A. Piros, L. Trautmann, and E. Baka, ‘‘Error handling method for digital
twin-based plasma radiation detection,’’ Fusion Eng. Design, vol. 156,
Jul. 2020, Art. no. 111592.

[115] M. G. Kapteyn, D. J. Knezevic, D. B. P. Huynh, M. Tran, and
K. E. Willcox, ‘‘Data-driven physics-based digital twins via a library of
component-based reduced-order models,’’ Int. J. Numer. Methods Eng.,
Jun. 2020.

[116] Y. Ye, Q. Yang, F. Yang, Y. Huo, and S. Meng, ‘‘Digital twin for the
structural health management of reusable spacecraft: A case study,’’ Eng.
Fract. Mech., vol. 234, Jul. 2020, Art. no. 107076.

[117] K. P. Murphy, ‘‘Dynamic Bayesian networks: Representation, inference
and learning, dissertation,’’ Ph.D. dissertation, Dept. Comput. Sci., Univ.
California, Berkeley Berkeley, CA, USA, 2002.

[118] P. E. Leser, J. E. Warner, W. P. Leser, G. F. Bomarito, J. A. Newman,
and J. D. Hochhalter, ‘‘A digital twin feasibility study (Part II): Non-
deterministic predictions of fatigue life using in-situ diagnostics and
prognostics,’’ Eng. Fract. Mech., vol. 229, Apr. 2020, Art. no. 106903.

[119] H. Zhang, Q. Yan, and Z. Wen, ‘‘Information modeling for cyber-physical
production system based on digital twin and automationml,’’ Int. J. Adv.
Manuf. Technol., pp. 1–19, Mar. 2020.

[120] Z. Liu, W. Chen, C. Zhang, C. Yang, and H. Chu, ‘‘Data super-network
fault prediction model and maintenance strategy for mechanical product
based on digital twin,’’ IEEE Access, vol. 7, pp. 177284–177296, 2019.

[121] W. Booyse, D. N. Wilke, and S. Heyns, ‘‘Deep digital twins for detection,
diagnostics and prognostics,’’ Mech. Syst. Signal Process., vol. 140,
Jun. 2020, Art. no. 106612.

[122] H. Kim, C. Jin, M. Kim, and K. Kim, ‘‘Damage detection of bottom-set
gillnet using artificial neural network,’’ Ocean Eng., vol. 208, Jul. 2020,
Art. no. 107423.

[123] W. Luo, T. Hu, C. Zhang, and Y. Wei, ‘‘Digital twin for CNC machine
tool: Modeling and using strategy,’’ J. Ambient Intell. Humanized Com-
put., vol. 10, no. 3, pp. 1129–1140, Mar. 2019.

[124] W. Luo, T. Hu, Y. Ye, C. Zhang, and Y. Wei, ‘‘A hybrid predictive
maintenance approach for CNC machine tool driven by digital twin,’’
Robot. Comput.-Integr. Manuf., vol. 65, Oct. 2020, Art. no. 101974.

[125] X. Song, T. Jiang, S. Schlegel, and D. Westermann, ‘‘Parameter tuning
for dynamic digital twins in inverter-dominated distribution grid,’’ IET
Renew. Power Gener., vol. 14, no. 5, pp. 811–821, Apr. 2020.

[126] S. K. Andryushkevich, S. P. Kovalyov, and E. Nefedov, ‘‘Composition
and application of power system digital twins based on ontological mod-
eling,’’ in Proc. IEEE 17th Int. Conf. Ind. Informat. (INDIN), vol. 1,
Jul. 2019, pp. 1536–1542.

[127] D. Gong, J. Sun, and Z. Miao, ‘‘A set-based genetic algorithm for interval
many-objective optimization problems,’’ IEEE Trans. Evol. Comput.,
vol. 22, no. 1, pp. 47–60, Feb. 2018.

[128] M. Zhou, J. Yan, and D. Feng, ‘‘Digital twin framework and its application
to power grid online analysis,’’ CSEE J. Power Energy Syst., vol. 5, no. 3,
pp. 391–398, 2019.

[129] C. C. Lee, ‘‘Fuzzy logic in control systems: Fuzzy logic controller. II,’’
IEEE Trans. Syst., Man, Cybern., vol. 20, no. 2, pp. 419–435, Apr. 1990.

[130] J. Morton, T. A. Wheeler, and M. J. Kochenderfer, ‘‘Analysis of recurrent
neural networks for probabilistic modeling of driver behavior,’’ IEEE
Trans. Intell. Transp. Syst., vol. 18, no. 5, pp. 1289–1298, May 2017.

[131] X. Chen, C. Wu, T. Chen, H. Zhang, Z. Liu, Y. Zhang, and M. Bennis,
‘‘Age of information aware radio resource management in vehicular
networks: A proactive deep reinforcement learning perspective,’’ IEEE
Trans. Wireless Commun., vol. 19, no. 4, pp. 2268–2281, Apr. 2020.

[132] L. Zhao, G. Han, Z. Li, and L. Shu, ‘‘Intelligent digital twin-based
software-defined vehicular networks,’’ IEEE Netw., vol. 34, no. 5,
pp. 178–184, Sep. 2020.

[133] B. R. Barricelli, E. Casiraghi, J. Gliozzo, A. Petrini, and S. Valtolina,
‘‘Human digital twin for fitness management,’’ IEEE Access, vol. 8,
pp. 26637–26664, 2020.

[134] O. Mazumder, D. Roy, S. Bhattacharya, A. Sinha, and A. Pal, ‘‘Synthetic
PPG generation from haemodynamic model with baroreflex autoregula-
tion: A digital twin of cardiovascular system,’’ in Proc. 41st Annu. Int.
Conf. IEEE Eng. Med. Biol. Soc. (EMBC), Jul. 2019, pp. 5024–5029.

[135] N. K. Chakshu, J. Carson, I. Sazonov, and P. Nithiarasu, ‘‘A semi-
active human digital twin model for detecting severity of carotid stenoses
from head vibration—A coupled computational mechanics and computer
vision method,’’ Int. J. Numer. methods Biomed. Eng., vol. 35, no. 5,
p. e3180, 2019.

[136] F. Laamarti, H. Faiz Badawi, Y. Ding, F. Arafsha, B. Hafidh, and
A. El Saddik, ‘‘An ISO/IEEE 11073 standardized digital twin frame-
work for health and well-being in smart cities,’’ IEEE Access, vol. 8,
pp. 105950–105961, 2020.

[137] N. S. Altman, ‘‘An introduction to kernel and nearest-neighbor non-
parametric regression,’’ Amer. Statistician, vol. 46, no. 3, pp. 175–185,
Aug. 1992.

[138] C. Cortes and V. Vapnik, ‘‘Support-vector networks,’’ Mach. Learn.,
vol. 20, no. 3, pp. 273–297, 1995.

[139] M. Pengnoo, M. Taynnan Barros, L. Wuttisittikulkij, B. Butler, A. Davy,
and S. Balasubramaniam, ‘‘Digital twin for metasurface reflector man-
agement in 6G terahertz communications,’’ IEEE Access, vol. 8,
pp. 114580–114596, 2020.

[140] R. Zhang, Y. Yang, W. Wang, L. Zeng, J. Chen, and S. McGrath, ‘‘An algo-
rithm for obstacle detection based on YOLO and light filed camera,’’ in
Proc. 12th Int. Conf. Sens. Technol. (ICST), Dec. 2018, pp. 223–226.

[141] G. Schrotter and C. Hürzeler, ‘‘The digital twin of the city of Zurich
for urban planning,’’ PFG, J. Photogramm., Remote Sens. Geoinf. Sci.,
pp. 1–14, Feb. 2020.

[142] H. Lehner and L. Dorffner, ‘‘Digital geoTwin Vienna: Towards a digital
twin city as Geodata Hub,’’ PFG, J. Photogramm., Remote Sensing
Geoinformat. Sci. volume, vol. 88, pp. 63–75, 2020.

[143] J. Döllner, ‘‘Geospatial artificial intelligence: Potentials of machine learn-
ing for 3D point clouds and geospatial digital twins,’’ PFG, J. Pho-
togramm., Remote Sens. Geoinformation Sci., pp. 1–10, 2020.

[144] X. Tong, Q. Liu, S. Pi, and Y. Xiao, ‘‘Real-time machining data appli-
cation and service based on IMT digital twin,’’ J. Intell. Manuf., vol. 8,
pp. 1–20, Oct. 2019.

[145] A. M. Lund, K. Mochel, J. Lin, R. Onetto, J. Srinivasan, P. Gregg,
J. E. Bergman, K. D. Hartling, A. Ahmed, and S. Chotai, ‘‘Digital system
and method for managing a wind farm having plurality of wind turbines
coupled to power grid,’’ U.S. Patent 10 132 295, Nov. 20, 2018.

[146] T. Shah, S. Govindappa, P. Nistler, and B. Narayanan, ‘‘Digital twin
system for a cooling system,’’ U.S. Patent 9 881 430, Jan. 30, 2018.

[147] H. Wang, ‘‘Digital twin based management system and method and
digital twin based fuel cell management system and method,’’ U.S. Patent
10 522 854, Dec. 31, 2019.

[148] J. E. Hershey, F. W. Wheeler, M. C. Nielsen, C. D. Johnson,
M. J. Dell’Anno, and J. Joykutti, ‘‘Digital twin of twinned physical sys-
tem,’’ U.S. Patent App. 15 087 217, Oct. 5, 2017.

[149] Z. Song and A. M. Canedo, ‘‘Digital twins for energy efficient asset
maintenance,’’ U.S. Patent App. 15 052 992, Aug. 25, 2016.

[150] C. J. Yates, M. Stankiewicz, J. Alexander, and C. Softley, ‘‘Industrial
safety monitoring configuration using a digital twin,’’ U.S. Patent App.
16 189 116, May 14, 2020.

[151] T. Masuda, B. Kim, and S. Shiraishi, ‘‘Proactive vehicle mainte-
nance scheduling based on digital twin simulations,’’ U.S. Patent App.
15 908 768, Aug. 29, 2019.

[152] H. Goldfarb, A. Pandey, and W. Yan, ‘‘Feature selection and feature
synthesis methods for predictive modeling in a twinned physical system,’’
U.S. Patent App. 15 350 665, May 17, 2018.

VOLUME 9, 2021 32051

M. M. Rathore et al.: Role of AI, Machine Learning, and Big Data in Digital Twinning: A SLR, Challenges, and Opportunities

[153] J. Zimmerman, C. Dodd, and M. Peterson, ‘‘Methods and systems for gen-
erating a patient digital twin,’’ U.S. Patent App. 15 635 805, Jan. 3, 2019.

[154] S. Nagesh, ‘‘X-ray tube bearing failure prediction using digital twin
analytics,’’ U.S. Patent 10 524 760, Jan. 7, 2020.

[155] M. Peterson, ‘‘Surgery digital twin,’’ U.S. Patent App. 15 711 786,
Mar. 21, 2019.

[156] L. G. E. Cox, C. P. Hendriks, M. Bulut, V. Lavezzo, and O. van der Sluis,
‘‘Digital twin operation,’’ U.S. Patent App. 16 704 495, Jun. 11, 2020.

[157] K. Fischer and M. Heintel, ‘‘Examining a consistency between reference
data of a production object and data of a digital twin of the production
object,’’ U.S. Patent App. 15 750 538, Aug. 9, 2018.

[158] M. G. Burd and P. F. McLaughlin, ‘‘Integrated digital twin for an indus-
trial facility,’’ U.S. Patent App. 15 416 569, Jul. 26, 2018.

[159] K. Deutsch, S. Pal, R. Milev, and K. Yang, ‘‘Contextual digital twin
runtime environment,’’ U.S. Patent 10 564 993, Feb. 18, 2020.

[160] S. Shiraishi, Z. Jiang, and B. Kim, ‘‘Digital twin for vehicle risk evalua-
tion,’’ U.S. Patent App. 16 007 693, Dec. 19, 2019.

[161] S. Shiraishi and Y. Zhao, ‘‘Sensor-based digital twin system for vehicular
analysis,’’ U.S. Patent App. 15 925 070, Sep. 19, 2019.

[162] A. Yousif, A. Ayyagari, D. T. Kirkland, E. C. Owyang, J. Apanovitch,
and T. W. Anstey, ‘‘Aircraft communications system with an operational
digital twin,’’ U.S. Patent App. 16 100 985, Feb. 13, 2020.

[163] Y. Park, S. R. Sinha, V. Venkiteswaran, V. S. Chennupati, and
E. S. Paulson, ‘‘Building system with digital twin based data ingestion
and processing,’’ U.S. Patent 10 854 194, Dec. 1, 2020.

[164] Q. Min, Y. Lu, Z. Liu, C. Su, and B. Wang, ‘‘Machine learning based
digital twin framework for production optimization in petrochemical
industry,’’ Int. J. Inf. Manage., vol. 49, pp. 502–519, Dec. 2019.

[165] S. Bangsow, Tecnomatix Plant Simulation. Springer, 2015.
[166] A. Glikson, ‘‘Fi-Ware: Core platform for future Internet applications,’’ in

Proc. 4th Annu. Int. Conf. Syst. Storage, 2011.
[167] A. Bosch Rexroth. Indramotion Mtx. Accessed: 2010.

https://www.boschrexroth.com/en/us/products/product-groups/electric-
drives-and-controls/topics/cnc/indramotion-mtx-standard-performance-
and-advanced/index

[168] T. White, Hadoop: The Definitive Guide. Newton, MA, USA:
O’Reilly Media, 2012.

[169] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin,
S. Ghemawat, G. Irving, M. Isard, and M. Kudlur, ‘‘TensorFlow: A sys-
tem for large-scale machine learning,’’ in Proc. 12th USENIX Symp.
Operating Syst. Design Implement. (OSDI), 2016, pp. 265–283.

[170] F. Seide and A. Agarwal, ‘‘CNTK: Microsoft’s open-source deep-learning
toolkit,’’ in Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discovery Data
Mining, Aug. 2016, p. 2135.

[171] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick,
S. Guadarrama, and T. Darrell, ‘‘Caffe: Convolutional architecture for
fast feature embedding,’’ in Proc. 22nd ACM Int. Conf. Multimedia,
Nov. 2014, pp. 675–678.

[172] A. Gulli and S. Pal, DeepLearning WithKeras. Birmingham, U.K.: Packt,
2017.

[173] M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and
I. H. Witten, ‘‘The WEKA data mining software: An update,’’ ACM
SIGKDD Explor. Newslett., vol. 11, no. 1, pp. 10–18, 2009.

[174] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman,
J. Tang, and W. Zaremba, ‘‘OpenAI gym,’’ 2016, arXiv:1606.01540.
[Online]. Available: http://arxiv.org/abs/1606.01540

[175] Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel, ‘‘Bench-
marking deep reinforcement learning for continuous control,’’ in Proc.
Int. Conf. Mach. Learn., 2016, pp. 1329–1338.

M. MAZHAR RATHORE (Member, IEEE)
received the master’s degree in computer and com-
munication security from the National University
of Sciences and Technology, Pakistan, in 2012,
and the Ph.D. degree in computer science and
engineering from Kyungpook National University,
South Korea, in 2018. He is currently working
as a Postdoctoral Researcher with the College of
Science and Engineering, Hamad Bin Khalifa Uni-
versity, Qatar. His research interests include big

data analytics, the Internet of Things, smart systems, network traffic analysis
and monitoring, remote sensing, smart city, urban planning, intrusion

detection, and information security and privacy. He is a professional member
of ACM. He received the Best Project/Paper Award in the 2016 Qual-
comm Innovation Award at Kyungpook National University, for his paper
‘‘IoT-Based Smart City Development Using Big Data Analytical Approach.’’
He was also a nominee for the Best Project Award in the 2015 IEEE
Communications Society Student Competition, for his Project ‘‘IoT-Based
Smart City.’’ He is serving frequently as a Reviewer for various IEEE, ACM,
Springer, and Elsevier journals.

SYED ATTIQUE SHAH received the Ph.D. degree
from the Institute of Informatics, Istanbul Techni-
cal University, Istanbul, Turkey. During his Ph.D.
degree, he was a Visiting Scholar with National
Chiao Tung University, Taiwan, The University of
Tokyo, Japan, and the Tallinn University of Tech-
nology, Estonia, where he completed the major
content of his thesis. He was an Assistant Professor
and the Chairperson of the Department of Com-
puter Science, BUITEMS, Quetta, Pakistan. He is

currently a Lecturer with the Data Systems Group, Institute of Computer
Science, University of Tartu, Estonia. His research interests include big data
analytics, cloud computing, information management, and the Internet of
Things.

DHIRENDRA SHUKLA is currently a Professor
and the Dr. J Herbert Smith ACOA Chair in tech-
nology management and entrepreneurship of the
University of New Brunswick (UNB), Canada.
He utilizes his expertise from the telecom sec-
tor and extensive academic background in the
areas of entrepreneurial finance, masters of busi-
ness administration, and engineering, to promote
a bright future for New Brunswick. Recogni-
tion of his tireless efforts and vision are demon-

strated through the UNB’s 2014 Award from Startup Canada as the ‘‘Most
Entrepreneurial Post-Secondary Institution of the Year,’’ his nomination as a
Finalist for the Industry Champion by KIRA, and his nomination as a Finalist
for the Progress Media’s Innovation in Practice Award. He was nominated for
the RBC Top 25 Canadian Immigrant Award and selected by a panel of judges
as a Top 75 finalist. Most recently, he received the Entrepreneur Promotion
Award by Startup Canada in 2017, as well as the Outstanding Educator
Award by the Association of Professional Engineers and Geoscientists of
New Brunswick in 2018.

ELMAHDI BENTAFAT received the bachelor’s
and M.Sc. degrees in computer science from the
Ecole Nationale Supérieure d’Informatique, Alge-
ria, in 2012 and 2016, respectively. He is currently
pursuing the Ph.D. degree with the College of Sci-
ence and Engineering, Hamad Bin Khalifa Univer-
sity, Qatar. His research interests include applied
cryptography, privacy, information security, and
network security.

SPIRIDON BAKIRAS (Member, IEEE) received
the B.S. degree in electrical and computer engi-
neering from the National Technical University of
Athens, in 1993, the M.S. degree in telematics
from the University of Surrey, in 1994, and the
Ph.D. degree in electrical engineering from the
University of Southern California, in 2000. He is
currently an Associate Professor with the College
of Science and Engineering, Hamad Bin Khalifa
University, Qatar. Before that, he held teaching

and research positions at Michigan Technological University, The City Uni-
versity of New York, The University of Hong Kong, and The Hong Kong
University of Science and Technology. His current research interests include
security and privacy, applied cryptography, mobile computing, and spa-
tiotemporal databases. He is a member of ACM. He was a recipient of the
U.S. National Science Foundation (NSF) CAREER Award.

32052 VOLUME 9, 2021

Journalof Data Science

355-376 , DOI: 10.6339/JDS.201804_16(2).0007

CAN EMOTICONS BE USED TO PREDICT SENTIMENT?

Keenen Cates1, Pengcheng Xiao1,∗, Zeyu Zhang1, Calvin Dailey1

1
Department of Mathematics, University of Evansville

1800 Lincoln Ave, Evansville, Indiana, 47722 USA

Abstract: Getting a machine to understand the meaning of language is a

largely important goal to a wide variety of fields, from advertising to enter-

tainment. In this work, we focus on Youtube comments from the top two-

hundred trending videos as a source of user text data. Previous Sentiment

Analysis Models focus on using hand-labelled data or predetermined

lexicon-s.Our goal is to train a model to label comment sentiment with

emoticons by training on other user-generated comments containing

emoticons. Naive Bayes and Recurrent Neural Network models are both

investigated and im- plemented in this study, and the validation accuracies

for Naive Bayes model and Recurrent Neural Network model are found to

be .548 and .812.

Key words: Sentiment analysis, Emoticons, Natural Language Processing,

Machine Learning.

1. Introduction

Sentiment analysis is a branch of natural language processing that involves trying to

understand the underlying sentiment and emotion behind language. For example,“Have a

great day” has a positive sentiment, and “Have a bad day” has a negative sentiment.

Current state of the art techniques for modelling sentiment in language involve using

machine learning and deep neural networks to classify the sentiment of language. For

example, SemEval is a yearly contest for trying to classify tweets as Positive, Negative, or

Neutral. Its findings advance the field of sentiment analysis and machine learning

(Rosenthal, Noura, and Preslav 2017).

356 Can emoticons be used to predict sentiment?

1.1 Objectives

Our focus is on another major social platform, Youtube, which garners hundreds of

thousands of comments and other user generated statistics. User data yields important

results in the fields of social sciences. In particular we are in- terested in the top trending

Youtube videos,and aim to identify sentiment of commenters by suggesting what emoticon

a user might use with their comments. We suggest emoticons give insight into the

sentiment of the user, and the emoticons pictographic nature gives us a better language to

indicate emotion. Using the subset of comments with emoticons we engineered a

labelled dataset of com- ments and emoticons. Our models take advantage of this

labelling to model the emoticon lexicon. This is further used to suggest what emoticons

might ac- company a comment (Hogenboom 2013). Using this dataset and the models we

have create, we hope to answer whether or not we can accurately predict what emoticon a

user might use.

1.2 Literature Review

Sentiment Analysis drives many industries and being able to correctly identif

y

sentiment in a Youtube comment would allow automated systems to moderate comments

or correctly recommend media or advertisements to users. In general, there are two

methods that Natural Language Processing researchers use for Sentiment Analysis;

Lexicon based and Machine Learning based. Sentiment Analysis is a fairly robust field,

and has consistently seen interest since its conception. This field has increased

exponentially with the surge in data seen with the rise of the internet, in many cases the

amount of data is intractable. Social platforms such as Youtube, by themselves generate

more data than any one hu- man could analyze. Therefore a system of Natural Language

Processing (NLP) is required to deal with the sheer volume of data.

Natural Language Processing can be considered a subset of cognitive science or

computer science. The concept of natural language processing originally came about in the

mid-20th century. The initial motivation was language translation (Salas-Za ŕate 2017).

Natural Language Processing naturally lends itself to the field of Artificial Intelligence, as

there is a strong desire for agents that can understand human language; for example, a chat

bot. Sentiment Analysis did not pull much attention until the early 2000s. The natural

language processing systems that were developed at first were only applicable to narrow

subject areas, such as answering questions with information from a database about moon

rocks, or answering questions from a manual on airplane maintenance (Liu 2012). The

Keenen Cates, Pengcheng Xiao, Zeyu Zhang, Calvin Dailey 357

explosion of social data quickly created a necessity to autonomously understand language

sentiment. Especially with the ubiquitous nature of social media in recent years, the field

of sentiment analysis has become more and more applicable to many fields. It has been

one of the most active areas of research in the field of natural language processing since

the turn of the century (Pozzi 2017).

There are many commercial applications. It may have significant effects for the fields

of management, political science, economics, and other social sciences, among others (Liu

2012). Sentiment analysis, also known as opinion mining, refers to the process of creating

automatic tools or systems which can derive subjective information from text in natural

(human) languages, as opposed to computer codes. The subjective information most

commonly desired by researchers are opinions and sentiments, hence the name sentiment

analysis. Sentiment analysis, while originally only practiced by computer scientists, has

become widely used by the management scientists and the social sciences. Microsoft,

Google, Hewlett-Packard, IBM, and others have created their own systems for sentiment

analysis.

Before the turn of the century, there were previous developments in what would later

become the field of sentiment analysis. Naive Bayes classifier pro- vided a way to model

the affective tone of an entire document based on the “semantic differential scores” of each

of the words in the document. The semantic meanings and scores were derived from a 1965

study by Heise. According to Lee and Pang (2002) marked an explosion of research in

sentiment analysis. This increase in the study of this topic was partially attributed to the

increasing popularity of machine learning models, and the availability of training sets with

which machine learning models could be trained. Turney (2002) used an algorithm based

on parts-of-speech tagging and semantic orientation in order to classify online reviews as

recommended or not recommended. Anderson and McMaster (1982) used machine

learning techniques such as Support Vector Ma- chines and Naive Bayes in order to

classify the sentiment of movie reviews. Dave, Lawrence, and Pennock (2003) classified

polarity of web reviews based on several n-gram methods. It was not as accurate when

applied to individual sentences because it was developed with the purpose of classifying

reviews which normally contained multiple sentences. Hu and Liu (2004) used a method

that could predict the sentimental orientation of opinion words and therefore the opinion

orientation of a sentence. It was an unsupervised method and did not require a corpus, and

was loosely based off the work of Dave, Lawrence and Pennock. It returned the

sentiments at the sentence level instead of at the entire review at once. Then it combined

the sentence-level sentiments to give a summary of the entire review. Moraes, Valiati, and

Neto (2013) showed the effectiveness of machine learning processes as opposed to

358 Can emoticons be used to predict sentiment?

lexicon-based models. They empirically compared the Support Vector Machines and

Artificial Neural Network machine learning methods for sentiment analysis and found that

the Artificial Neural Networks performed better. In 2015, Wang, Liu, Sun, Wang.B, and

Wang.X. showed the effectiveness of Long short-term memory recurrent neural networks

for sentiment analysis by predicting the sentiments of tweets.

1.3 Sentiment Lexicon

The lexicon method splits input text into many individual words or phrases called

tokens. Then, it creates a table of these tokens and records the number of times each token

shows up in the text. The resulting tally is called a “Bag of Words” model. Once this

process is done, another tool called “Sentiment Lexicon” is used for computing the

classification of the bag of tokens we mentioned above. The Sentiment Lexicon has the

sentiment values, which can be just positive or negative numbers or some other value-

representations, like vectors, that are pre-recorded for each token. This can be done either

manually or by some machine learning techniques. Once we have the input text tokenized

and a suit- able Sentiment Lexicon, the final task is to design a function to compute the

final sentiment. The simplest way to compute the final sentiment is to sum the sentiment

values of each token together. The lexicon method is a traditional way to deal with natural

language processing problems, and it has a good theoretical basis. Many people are still

using and studying this method in spite of its origins in the 1960s. However, it does have

some drawbacks such as ignoring the importance of integrality and continuity of the text.

We know that the meaning of a sentence highly depends on the order of words and context;

these should not be ignored if we want a real intelligent sentiment processing system

(Tbboada 2011).

1.4 Machine Learning

In the Machine Learning technique of sentiment analysis the classification algorithm

uses a training set to learn a model based on features in the set. This makes a more nuanced

classification possible and can help with ambiguous words or interpretations that vary by

context. A method of feature extraction must be chosen. Some of these methods include

n-grams, which are sets of words that contain n words each. Others use parts-of-speech

information, emotional, affective, or semantic data. One of the disadvantages of the

machine learning method is that it requires a large set of labelled data to be used as the

Keenen Cates, Pengcheng Xiao, Zeyu Zhang, Calvin Dailey 359

training set. It is simpler to use the lexicon-based method unless a suitable training set is

available (Salas-Za ŕate 2017).

We will need to classify the sentiments of the emoticons manually in order to prepare

them for use in our analysis. Once that is done, we can compile our training set using the

comments in the data that already contain emoticons, using the sentiments of each

emoticon. Then our model will be able to classify and assign an emoticon to each comment

in the data set that does not already contain one. Recurrent Neural Networks(RNNs) have

had a great deal of success in the Natural Language Processing Realm. The reason is that

text data is highly sequential, for example, the word “day” does not mean much unless you

know the words that came before it; i.e “Have a great day.” RNNs have pushed the state

of the art of previous architectures in short-length text data (Lee and Dernoncourt 2016).

Given previous attempts to model sentiment have not thoroughly explored emoticons,

we hope to answer the question of whether or not we can accurately recommend emoticons

that might accompany a piece of text. Once we have answered this, further research can

make attempts to analyze sentiment with emoticons on a machine.

2. Methodology

2.1 Data

To get our data, we used the Data Science Competition Website Kaggle. On this

website, people share datasets, competitions, and tutorials. We found a dataset containing

comments from the top 200 trending Youtube videos. The author of this dataset obtained

the data through Youtube’s publicly available API, which allows developers to easily

query for data on Youtube. The data itself contains profanity, nonsensical text, and in

general is noisy. The data itself could be generated by bots, and we do no vetting to

determine whether a comment actually comes form a human. The noisiness of the data

might prevent us from training a successful model; however, we assume that the large

amount of data will help our models perform well in spite of the low quality of data.

In order to answer the question of whether or not a model could recommend emoticons,

we created 2 models that attempt to perform this recommendation. We also created a

simple dummy model for purposes of comparison. We have roughly three-hundred

thousand comments with emoticons, and use that to boos- trap a dataset of comments with

labels. More data is desirable, but this is a fairly large corpus for initial research.

In total, there are 691, 388 rows in the dataset. A large proportion of them contain

emoticons, (more than 200, 000), so there is a quite a bit of data, and it would be fairly

360 Can emoticons be used to predict sentiment?

straightforward to access the Youtube API and get more if needed. This means I have as

much data as I could possibly want, and more if needed. As for features, I will only use

the text, likes, reply threads, and so on will be ignored in this phase of the project. On

average, each text is 15 words long. Figure 1 shows some examples of how the data looks.

Figure 1: Example unprocessed data

2.2 Evaluation Metrics

The models will be evaluated using a holdout set of data, in which each will

recommend five emoticons that might accompany a text. If at least one recommendation

is an emoticons that occurs in the validation comments, then I will consider it to be

a ”correct” guess. Accuracy is then the number of correct guesses divided by total guesses.

Keras calls this accuracy ”top k categorical accuracy”, and will be implemented for our

models. Mathematically, this would look something like this where matching x ∈

Comments and y ∈ Labels and score(x) = 1 if any p ∈ argmaxk=5(predict labels(x)) is in

y, else score(x) = 0. predict labels(x) would return the probabilities of each output class

occurring. Then the accuracy of the model would be
ΣN(score(xi))

?
where xi∈ Comments

and N =| Comments |.

One consideration is that the distribution of emoticons occurring in the corpus of data

is highly skewed; this would be good reason to suggest F1 scores and might be better for

future analysis. However, we chose this evaluation metric because it more closely

resembles the question we are asking. The important thing to note is that the distribution

is indeed skewed(see Figure 2).

Keenen Cates, Pengcheng Xiao, Zeyu Zhang, Calvin Dailey 361

Figure 2: Distribution of a subset of Emoticons

2.3 Analysis Plan

In order to compare the performance of our model, we created a holdout set of data

meant for only validation of accuracy. We also defined what a prediction would be for

each model, each model would output its top five highest predictions. If any of those

predictions are in the output validation set, then we considered it an accurate prediction.

Then in order to analyze the dataset, we will compute the prediction accuracy of each

model and compare those scores. One might also consider looking at the training accuracy

of each model; however, these scores are not directly comparable, so we ignore them

except for the purposes of optimizing the model.

2.4 Approach

In our approach, we had to make a few crucial assumptions and simplifications to

contextualize our problem. Firstly, our dataset involved input data with multiple output

classifications. For example, a users can add hundreds of the same emoticon or many

different emoticons. As a preprocessing step, we narrowed down these classes to the

unique emoticons that show up in a comment, and unrolled the data set to have a single

label. Table 1, displays how each comment gets unrolled into multiple data points with

single labels.

362 Can emoticons be used to predict sentiment?

I loved this video! x x y

I loved this video!

I loved this video!

x

y

Table 1: Unrolling of data labels

The other assumption exists only for our Naive Bayes Model, and it is that all words

in the comments are independent. This assumption is difficult to back up, and it is not clear

whether there is mutual dependence or mutual exclusivity between words. However, our

Recurrent Neural Network does not have this limitation because it can model the entire

sequence.

2.5 Preprocessing

One of the most important steps is the preprocessing stage. This is done before all

models are trained. We first separate the data into comments with emoticons and comments

without emoticons. We then make all comments lowercase and afterwards normalize our

comments on both by creating a dictionary of punctuation to tokens, and a dictionary of

word counts over all comments that use thes ordering of each word as its embedding. Table

2 shows an example of how the dictionaries are used to tokenize a comment. A similar

process is used to encode the emoticons, we use a dictionary to encode them as integers.

Preprocessing the comments in this way gives us a normalized integer sequence, which

deals with comments that might have different capitalizations of words.

2.6 Dummy Model

For purposes of comparison, we created a very simple model that always predicts that

a comment would use the emoticon with the largest prior probability. The motivation

behind this, is that it gives us a baseline score to beat. If we can do significantly better than

this, then we know that the models have potential.

2.7 Naive Bayes Model

Our second model uses Bayesian Statistics that creates tables of posterior proba-

bilities for each class given a word using Bayes rule. Naive Bayes is a conditional

probability model, and given some instance to be classified, represented by a vector of

features:

x = (?1,…,??)

We then compute the probability of each output class using conditional probability

Keenen Cates, Pengcheng Xiao, Zeyu Zhang, Calvin Dailey 363

p(??|?1,…,??)

Table 2: Tokenization Process

Since n, can be large making this model less tractable we need to reformulate our

model using Bayes Rule. In plain english,

????????? =
????? ∙ ??????ℎ???

????????

And symbolically,

p(??|?) =
?(??)?(?|??)

?(?)

In practice, the numerator is the most import part as the denominator does not depend

on effectively making it a constant. The numerator is equivalent to the joint probability

model meaning we can replace the numerator with,

p(??,?1,…,??)

We can then rewrite the numerator using the chain rule for repeated applications of

conditional probability, derivation is in appendix 1. Then we add the naive as- sumption

of conditional independence, allowing use to further simplify our model

364 Can emoticons be used to predict sentiment?

Figure 3: Naive Bayes Model

to:

p(??,?1,…,??) =
1

?
?(??)∏?(??|??)

?

?=1

Where Z is:

Z = p(x) = ∑?(??)
?

?(?|??)

Which is the scaling factor dependent on the instance. The derivation is in appendix 2.

In order to make a classifier, we would generally take the argmax of the simplified model

without Z, but in our case we take the top five arguments as our program is recommending

multiple emoticons that might be appropriate to the definition of Naive Bayes classifier .

We implement this model in python and the model follows figure 3.

Another problem is that we have to deal with words that never show up in our corpus

of texts. In order to deal with this, we smooth the probabilities. To do this, we make any

word or class that doesn’t show up have a very small probability that is close, but not zero.

Otherwise, the probability would zero out when words are not in the corpus.

2.8 Recurrent Neural Network

Our third and final model, is a recurrent neural network and our architecture is as

follows in table 3.

Keenen Cates, Pengcheng Xiao, Zeyu Zhang, Calvin Dailey 365

Input

Embedding Layer

LSTM Layer

LSTM Layer
LSTM Layer

Fully-Connected Layer

Output layer

Table 3: RNN Architecture

Recurrent Neural networks are a class of neural networks that form a directed cycle,

allowing them to take time into account, or a notion of memory. This allows for the

RNN

to be suited to predicted arbitrary sequences by taking advantage of their memories.

The label data also undergoes another transformation before the RNN begins the

learning process. Since the emoticons are encoded using an ordinal number, the integer

representation does not quite make sense as one emoticon is not greater than another. To

rectify this, we represent this integer as a one-hot vector, essentially we take a fixed-length

vector that is the size of the total number of output classes. Then the integer is used as an

index of the “hot” class. Table 4 gives a small example of encoding a small class space.

Table 4: One-Hot Encoding

One of the major features of this model is the stacked LSTM layers. This architecture

allows us to better model hierarchical elements of language. This means each layer will

represent progressively complex parts of the hierarchy. One might imagine this in terms

of the composition of the human face. For example, the most basic element is an edge.

Then a more complex step would be individual elements of the face such as a nose or

mouth. Then the most complex part would be the entire face, and the composition of its

requisite parts.

366 Can emoticons be used to predict sentiment?

The LSTM itself is able to remember previous contexts in sentences, meaning we

could potentially get more performance via our model becoming better at modelling

context.Our RNN had a much longer time to run, and in order to train the model, we

decided to use more power hardware in the form of a GPU. The Neural Network was then

trained on a GPU using Floyd Hub, a platform for running deep learning projects. The

expense was roughly 14 dollars, as a we subscribed to the Data Science plan which gave

us 10 hours of gpu time which we used for experimentation on multiple occasions. The

price was remarkably cheap compared to other platforms such as Amazon. Usage of

FloydHub is remarkably simply, and resembles version control programs such as git. One

simply uploads their code to the website using command line tools, and are given an

interface to interact with their instance. This service was worthwhile to learn because it

abstracted away elements such as infrastructure, version control, and storage and we could

focus on the problem.

In addition to our baseline architecture, we also preform dropout on each lay- er,

which helps prevent against training bias because the network probabilistic “drops” some

of the weight which forces the network to build redundancies. For the training metric, we

implemented the top k categorical accuracy metric listed in the evaluation metrics. For

the objective function we found that categorical cross entropy work best which typically

works well in multi-class, single-label s- cenarios.Using TFLearn, a deep learning library

for Python, we implemented the architecture we decided on with relative ease. TFLearn

builds on top of Tensor- Flow, abstracting away many of the more intimate computational

components, and allowing the programming to think about the layers and interactions

between layers rather than how to build a well known type of layer or cell.

2.9 Implementation

2.9.1 rogramming Language Libraries

•Python 3

•TFLearn a deep learning library featuring a higher-level API for Tensor- Flow.

•TensorFlow a deep learning library

As mentioned throughout the text, the models where implemented using the listed

libraries. We did our coding on the website FloydHub via iPython Notebooks, which

abstracted away much of the setup. We split our code up into three notebooks, one for

preprocessing, Bayesian Model, and RNN. We ran into very few problems implementing

our solution; however, some are outlined below.

Keenen Cates, Pengcheng Xiao, Zeyu Zhang, Calvin Dailey 367

2.9.2 Problems

•Bayes Smoothing We ran into a small hitch with the Bayesian when dealing with

querying prior probabilities when certain values did not exist in the data. However, we

used a technique to ”smooth” the values by assigning a small probability to these values.

•Skin Tone Modifters There are emoticons that exist that modify other emoticons, i.e.

allowing one to change the skin tone of the smiley face. We found that these confounded

our predictions, and removed them as possible predictions.

•Finding loss, activation, and metrics We had to experiment many times to find the

best loss, activation, and metric functions for our RNN. This process may be simple trial-

and-error as we experienced.

2.10 Reftnement

Originally, our RNN model did not preform as well as we had hoped; however, a few

optimization to our model vastly impacted our performance. The first model we used was

a multi-class, multi-label classifier which performed very poorly. Our RNN had

performance at .508 which left much to be desired. We believe the reason for this is that

instead of one-hot encoded vector, we had many-hot encoded. This means that the label

space would be of order 2# of emoticons. Since this space is extremely large, the model would

have trouble representing any reasonable portion of this. For this reason, we needed to

unroll data points to preform multi-class, single-label classification. After adjusting our

loss function, metric function, and activation function we ended up with much better

performance. We believe this to be because of the reduction in potential labels to just #

of emoticons. In addition, hyper parameters were adjusted, such as, learning rate and batch

size to find out what setting worked best. The best we found was a learning rate of .001

and a batch size of 128.

3. Results

In order to validate the models, we created a holdout set of labelled data that none of

the models got to use for training or testing. The accuracy of each model using top k

categorical accuracy is in tables 5 and 6.

368 Can emoticons be used to predict sentiment?

Model Accuracy

Dummy

Naive Bayes

RNN

.527

.859

.702

Table 5: Training Accuracy Results

Model Accuracy
Dummy
Naive Bayes
RNN
.527

.548

.812

Table 6: Validation Accuracy Results

Table 6 gives us a measurement of how well our recommendation engine gives us

accurate emoticons to represent our text. Our results do not promote strong confidence in

our Naive Bayes Model’s ability to recommend emoticons; however, there are some

potential improvements to the model such as n-gram modelling. Notably, the Bayesian

Model preforms decently on the training data, but generalizes quite poorly and shows signs

of over-fitting. The RNN on the other hand, surprisingly preforms slightly worse on

training, but preforms much better on the validation set. For whatever reason this

phenomenon occurs, it is clear that the model generalizes much better.

3.1 Visualization of Model Functionality

We have a model that could be incorporated into a wide variety of applications; for

example, a browser plugin that predicts what emoticons you might put with a comment

and assist the user similar to an auto-complete feature. One issue to consider might be the

nature of Youtube comments themselves, which might pre- vent the generalization of this

model to other applications. However, the models do show that this sort of functionality is

possible. For example, we have pulled some examples from the data and run them through

our models to produces the tables below, and the comments themselves seem to be quite

different than more formal forms of language.

Keenen Cates, Pengcheng Xiao, Zeyu Zhang, Calvin Dailey 369

Table 7: Example data and predictions

While the machine learning back-end may not be the most sophisticated, the model

does a good job in practice of giving recommendations, and we think the model would be

good enough to use for applications to be built on top of.

3.2 Limitations

One limitation of our models is that words that do not show up in the Youtube

Comment corpus cause issues, as our models have trouble predicting outputs for words

that it has never seen. One way to fix this, might be to mine for more Comment data. Some

drawbacks of the Naive Bayes Model is that we may not be able to model longer term

trends in comments, however with the short length of the comments, this may be a non

issue. We also are limited in our choice of language modelling because we are on the word

level. We would likely see large improvement by expanding our level of modelling to some

type of n-gram. The RNN has limitations in multi-class classification, and this may be

hindering its ability to learning. Another limitation might be that the training time is cost

prohibitive. The model would likely continue to learn and perform better with more

370 Can emoticons be used to predict sentiment?

training time and data, meaning ultimately a higher cost for the model. The naive bayes is

easy to program with fast run time, and no need to train for hours upon hours.

Another major consideration is that an RNN might be a bad fit. We originally though

long term sequential modelling would be important, but it turns out the average comment

length is 15 words long. It may be the case that sense the length of texts are so short, that

we might have to thoroughly rethink what our strategy would be if this sequential

modelling is unimportant.

3.3 Future Work

In order to eliminate the assumption of independence in the Bayesian model, we can

add complexity by changing at what level we model the data. To do such we

would need

to employ a skip-gram or n-gram model that contain larger parts of the sequence data. One

might also explore alternative Bayesian Models such as Hidden Markov Models. The same

improvements to the data modelling using n-grams would likely improve the quality of the

RNN results. The RNN model likely has a great deal of room for improvement, one might

experiment with hyperparameter tuning or modifying the architecture. There are even more

powerful models such as CRNNs and GANs that push the state of the art in deep learning.

These models would be worth exploring; however, we pushed our newfound deep learning

knowledge as far as we could in the time allotted.

Another important consideration is the unrolling of the data. Future work should

further explore how to deal with multi-class classification, which would likely involve

writing new validation and loss functions for the neural network model. However, the

Naive Bayes Model does not suffer from this limitation.

Future work might also try and further connect the emoticons and sentiment. We

hypothesize that emoticons will naturally lend themselves to a easily convert into

sentiment classes. However, our current models predict only what emoticon might be used,

and the user of the model would have to infer what sentiment the emoticon might convey

depending on context.

One might also find more optimizations by adding further preprocessing steps, for

example, eliminating common english words that add very little information.

Keenen Cates, Pengcheng Xiao, Zeyu Zhang, Calvin Dailey 371

3.4 Reflection

Looking back at the process, here are the steps we took to get to the current models

• Literature Review We made sure to have a rough idea of what people in this field

have tried, and what the state of the art is.

• Deciding on a Model After reviewing the field, we made a decision on what models

we wanted to implement which set the tone for preprocessing and implementation.

• FloydHub Next we setup our programming environment with cloud computing in

mind. It’s important to setup an environment such as FloydHub or AWS to minimize

training time on a fast gpu. At this step we also made sure to download all the libraries we

would need

• Preprocessing a large majority of time was spent trying to learn how to deal with

the data, and exploring the data itself. We had to go through multiple iterations of

embedding and tokenization to find the method that made sense.

• Model Implementation After preprocessing our data, this step was fairly

straightforward. Most of the time at this step is dealing with edge cases, or optimization of

models rather than the actual implementation.

• Reftnement Refinement may have been the hardest part because we had to make

inferences about why our model was not performing up to our desires. It’s hard to say what

the potential of each model was, so we kept iterating until we had something that seemed

substantial.

3.5 Conclusion

Overall, there are many areas for potential improvement, and our work serves as a

baseline for recommending emoticons. However, we have begun to answer our original

question, it seems plausible the emoticons can be assigned with accuracy to comments as

noisy as Youtube comments, making it easy for a casual observer to understand the

sentiment of a text.

Acknowledgment: The authors appreciate the anonymous referee for the con-

structive review of the paper which has greatly improve the quality of the article. The

authors would also like to thank the generous support from the mathematics department at

University of Evansville.

372 Can emoticons be used to predict sentiment?

Appendix

1. Chain rule for repeated applications of conditional probability.

p(??,?1,…,??) = ?(?1,…,??,??)

= ?(?1|?2 …,??,??)?(?2 …,??,??)

= ?(?1|?2 …,??,??)?(?2|?3 …,??,??)?(?3 … ,??,??)

=….

= ?(?1|?2 …,??,??)?(?2|?3 …,??,??)…?(??−1|??,??)?(??|??)p(??)

2. Naive Assumption of conditional independence to simplify model. This the joint

model can be derived via:

p(??|?1,…,??) = p(??,?1,… ,??)

= p(??)?(?1|??)?(?2|??)?(?3|??)…

= p(??)∏?(??|??)

?
?=1

Keenen Cates, Pengcheng Xiao, Zeyu Zhang, Calvin Dailey 373

References

[1] Anderson, C., McMaster, G. (1982). Computer Assisted Modeling of Affective Tone

in Written Documents. Computers and the Humanities, 16(1), 1-9.

[2] Brill, E., Mooney, R. J. (1997). An Overview of Empirical Natural Language

Processing. AI Magazine, 18(4), 13.

[3] Chipman, S. E. (2017). The Oxford Handbook of Cognitive Science. Oxford: Oxford

University Press.

[4] Dave, K., Lawrence, S., and Pennock, D. (2003). Mining the Peanut Gallery: Opinion

Extraction and Semantic Classification of Product Reviews. In Proceedings of the

12th International Conference on World Wide Web (WWW 03). ACM, New York,

NY, USA, 519-528.

[5] Hu, M., Liu, B. (2004). Mining and Summarizing Customer Reviews. In Pro- ceedings

of the Tenth ACM SIGKDD International Conference on Knowl- edge Discovery and

Data Mining pp. 168-177. ACM.

[6] Hogenboom, Alexander, et al.(2013) Exploiting emoticons in sentiment analysis.

Proceedings of the 28th Annual ACM Symposium on Applied Computing. ACM.

[7] Kang, Mangi, Jaelim Ahn, and Kichun Lee.(2017) Opinion mining using ensem- ble

text hidden Markov models for text classification. Expert Systems with Applications.

[8] Lee, Ji Young, and Franck Dernoncourt.(2016) Sequential short-text classifi- cation

with recurrent and convolutional neural networks. arXiv preprint arXiv:1603.03827.

[9] Liu, B. (2012). Sentiment Analysis and Opinion Mining. Morgan and Claypool.

LSTM Networks for Sentiment Analysis. DeepLearning 0.1 Documentation,

Deeplearning. Retrieved December 01, 2017.

[10] Mitchell, J. (Datasnaek). Trending Youtube Video Statistics and Comments.

Kaggle, Kaggle Inc., Aug./Sep. 2017.

374 Can emoticons be used to predict sentiment?

[11] Moraes, R., Valiati, J. F., Neto, W. P. G. (2013). Document-level Sentiment

Classification: An Empirical Comparison between SVM and ANN. Expert Systems

with Applications, 40(2), 621-633.

[12] Naive Bayes classifier.Wikipedia, Wikimedia Foundation INC, 30 Nov. 2017,

Available from http://en.wikipedia.org/wiki/NaiveBayesclassifier.

[13] Pang, B., Lee, L., Vaithyanathan, S. (2002). Thumbs up? Proceedings of the ACL-02

Conference on Empirical Methods in Natural Language Processing

– EMNLP 02.

[14] Pozzi, F. A. (2017). Sentiment Analysis in Social Networks. Amsterdam: Else- vier.

[15] Rosenthal Sara, Noura Farra, and Preslav Nakov. (2017 )SemEval-2017 task 4:

Sentiment analysis in Twitter. Proceedings of the 11th International Workshop on

Semantic Evaluation .

[16] Salas-Za ŕate, M. P., Medina-Moreira, J., Lagos-Ortiz, K., Luna-Aveiga, H.,

Rodr íguez-Garc ía, M. A .́, and Valencia-Garc ía, R.(2017) Sentiment Analy- sis on

Tweets about Diabetes: An Aspect-Level Approach. Computational and

Mathematical Methods In Medicine, 1-9.

[17] Siersdorfer, Stefan, et al.(2010) How useful are your comments?: analyzing and

predicting Youtube comments and comment ratings. Proceedings of the 19th

international conference on World wide web. ACM.

[18] Taboada, Maite, et al.(2011) Lexicon-based methods for sentiment analysis.

Computational linguistics 37.2:267-307.

[19] Turney, P. D. (2002). Thumbs Up or Thumbs Down?: Semantic Orientation Applied

to Unsupervised Classification of Reviews. In Proceedings of the 40th Annual

Meeting on Association for Computational Linguistics, pp. 417-424. Association for

Computational Linguistics.

[20] Wang, X., Liu, Y., Sun, C., Wang, B., Wang, X. (2015). Predicting Polarities of

Tweets by Composing Word Embeddings with Long Short-Term Memory. In

Proceedings of the 53rd Annual Meeting of the Association for Compu- tational

Linguistics and the 7th International Joint Conference on Natural Language

Keenen Cates, Pengcheng Xiao, Zeyu Zhang, Calvin Dailey 375

Processing Volume 1: Long Papers, pp. 1343-1353, Beijing, Chi- na. Association for

Computational Linguistics.

[21] Whitelaw, C., Garg, N., Argamon, S. (2005). Using Appraisal Groups for Sen- timent

Analysis. In Proceedings of the 14th ACM International Conference on Information

and Knowledge Management, pp. 625-631. ACM.

Keenen Cates1, Pengcheng Xiao1,∗, Zeyu Zhang1, Calvin Dailey1

1Department of Mathematics, University of Evansville

1800 Lincoln Ave, Evansville, Indiana, 47722 USA

∗Corresponding author: px3@evansville.edu; fax: (812)488-2944

Copyright of Journal of Data Science is the property of National University of Kaohsiung,
Department of Applied Mathematics and its content may not be copied or emailed to multiple
sites or posted to a listserv without the copyright holder’s express written permission.
However, users may print, download, or email articles for individual use.

Received July 1, 2017, accepted August 7, 2017, date of publication August 18, 2017, date of current version October 25, 2017.

Digital Object Identifier 10.1109/ACCESS.2017.2740982

A Pattern-Based Approach for Multi-Class
Sentiment Analysis in Twitter
MONDHER BOUAZIZI AND TOMOAKI OHTSUKI, (Senior Member, IEEE)
Graduate School of Science and Technology, Keio University, Yokohama 223-8522, Japan

Corresponding author: Mondher Bouazizi (bouazizi@ohtsuki.ics.keio.ac.jp)

ABSTRACT Sentiment analysis and opinion mining in social networks present nowadays a hot topic of
research. However, most of the state of the art works and researches on the automatic sentiment analysis
and opinion mining of texts collected from social networks and microblogging websites are oriented toward
the binary classification (i.e., classification into ‘‘positive’’ and ‘‘negative’’) or the ternary classification
(i.e., classification into ‘‘positive,’’ ‘‘negative,’’ and ‘‘neutral’’) of texts. In this paper, we propose a novel
approach that, in addition to the aforementioned tasks of binary and ternary classifications, goes deeper in the
classification of texts collected from Twitter and classifies these texts into multiple sentiment classes. While
in this paper, we limit our scope to seven different sentiment classes, the proposed approach is scalable and
can be run to classify texts into more classes. We first introduce SENTA, our tool built to help users select
out of a wide variety of features the ones that fit the most for their application, to run the classification,
through an easy-to-use graphical user interface. We then use SENTA to run our own experiments of multi-
class classification. Our experiments show that the proposed approach can reach up to 60.2% accuracy on
the multi-class classification. Nevertheless, the approach proves to be very accurate in binary classification
and ternary classification: in the former case, we reach an accuracy of 81.3% for the same data set used after
removing neutral tweets, and in the latter case, we reached an accuracy of classification equal to 70.1%.

INDEX TERMS Twitter, sentiment analysis, machine learning.

I. INTRODUCTION
Twitter, as well as other Online Social Networks (OSN) and
microblogging websites became literally the biggest web
destinations for people to communicate with each other, to
express their thoughts about products [1], [2] or movies [3],
share their daily experience and communicate their opin-
ion about real-time and upcoming events, such as sports or
political elections [4], etc.

While new platforms such as Snapchat1 focused on video-
and multimedia-based communication, Twitter kept some
properties that make it a very interesting subject of data
mining:
• In its basic form, Twitter is a microblogging service that
allows users to post brief text updates, with the unique
property of not allowing more than 140 characters in
one text message. This limitation turned out to be a very
attractive property, since it allows posting quick, even
real-time, updates regarding one’s activities and facil-
itates sharing and forwarding status messages, as well
as replying to them quickly [5]. This allows the quick
spread of news or information, regardless of whether that

1https://www.snapchat.com

has a positive impact or a negative one, whether the news
spread are correct or false, etc.

• The openness and ease of access to posts from different
sources attract people more than ever: while in most of
the cases, the access to someone’s updates requires a
mutual friendship on OSNs such as Facebook, Twitter
allows any user to follow another, without the reciprocal
being true.

• The wide use of hashtags makes it easy for people
to search for tweets dealing with a specific subject.
Hashtags are labels ‘‘used on social network and
microblogging services which makes it easier for users
to find messages with a specific theme or content’’.2

Hashtags also allow users to categorize their own tweets
so that other users know what the tweet is dealing with.

Thanks to these properties, this ecosystem presents a very
rich, source of data to mine. However, due to the limitation
in terms of characters (i.e. 140 characters per tweet), mining
such data present lower performances than that when mining
longer texts. In addition, classification into multiple classes
remains a challenging task: binary classification of a text

2https://en.wikipedia.org/wiki/Hashtag

VOLUME 5, 2017
2169-3536
2017 IEEE. Translations and content mining are permitted for academic research only.

Personal use is also permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

20617

M. Bouazizi, T. Ohtsuki: Pattern-Based Approach for Multi-Class Sentiment Analysis in Twitter

usually relies on the sentiment polarity of its components
(i.e., whether they are positive or negative); whereas, when
positive and negative classes are divided into subclasses, the
accuracy tends to decrease remarkably.

In this paper, we propose an approach that relies on writing
patterns, and special unigrams to classify tweets into 7 dif-
ferent classes, and demonstrate how the proposed approach
presents good performances (i.e., classification accuracy and
precision). The main contributions present in this paper are
as follows:

1) We introduce SENTA, a user-friendly tool that allows
the extraction of a wide set of features from texts that
cover both the content and the form,

2) We introduce, in addition to some conventional fea-
tures, writing pattern-related features to help enhance
the accuracy of classification,

3) We use SENTA to extract an optimal set of features to
classify tweets into 7 different sentiment classes.

The remainder of this paper is structured as follows:
In Section II, we present our motivations for this work
and in Section III, we describe some of the related work.
In Section IV, we present SENTA, our tool to extract different
features from tweets, and that we will use later on to per-
form the multi-class classification. In Section V, we describe
in details the proposed method. In Section VI, we detail
our experiments and the results obtained. Section VII con-
cludes this paper and proposes possible directions for future
work.

II. MOTIVATIONS
A. WHY MULTI-CLASS SENTIMENT ANALYSIS?
Social networks and microblogging websites such as Twitter
have been the subject to many studies in the recent few years.
Automatic sentiment analysis and opinion mining present a
hot topic of study. Social networks present a huge source
of data representing the opinions of a significant, yet totally
random, proportion of users and customers who are using a
product of a service. However, due to the informal language
used, the presence of non-textual content and the use of slang
words and abbreviations, classification of data extracted from
such microblogging websites is rather a challenging task.
Ghag and Shah [6] defines ‘‘Hidden Sentiment Identifica-
tion’’ which is the identification of the real feeling rather
than the sentiment polarity, ‘‘Handling Polysemy’’ which is
the existence of multiple meanings that might have different
sentiment polarity for the same word, and ‘‘Mapping Slangs’’
which is the identification of the meaning and the polarity
of slang words, among others as the most challenging tasks
facing the sentiment analysis of short microblog texts.

On a related context, the state of the art proposed
approaches are mostly focusing on the binary and ternary sen-
timent classification. In other words, they classify texts either
into ‘‘positive’’ and ‘‘negative’’, or into ‘‘positive’’, ‘‘nega-
tive’’ and ‘‘neutral’’. However, to study the opinion of a user,
it would be more interesting to go deeper in the classification,
and detect the sentiment hidden behind his post. Following

two examples of tweets which are negative, however, reflect
two completely different aspects:
• ‘‘Damn damn.. no iPhone support for windows XP x64.
There are some workarounds, but I can’t figure this out.’’

• ‘‘Nooooooooooo! My iPhone glass cracked :(’’
In the first example, the user is expressing his fury towards
the absence of support of his phone on an operating system.
However, in the second, the user is expressing some feeling
of sadness because of a physical problem his phone faced.
The first example shows some important information regard-
ing the satisfaction of the user, therefore, it might be more
important to study. However, in general, both information can
be used, yet, they have to be distinguished from each other.

B. THE NEED FOR AN OPEN-SOURCE TOOL FOR
FEATURE EXTRACTION FROM TWEETS
Nowadays, a variety of tools such as LIWC [7] offer the
option to extract advanced features for different languages
from texts, most of these tools are paid and require some
programming knowledge to use.

In addition, to the best of our knowledge, none of these
tools offer the possibility to extract, in a flexible way, writing
patterns, that can be used to enhance the performances of
classification tasks such as the detection of sarcasm or, as in
the current work, the multi-class sentiment analysis.

Therefore, arises the need for a more flexible, yet easy
to use and user-friendly tool that allows the extraction of
multiple types of features, while offering the possibility to
customize them depending on the use case, to obtain perfor-
mances as high as possible.

In this work, we present SENTA, an open-source tool that
performs the extraction of features and save them either in an
excel format sheet or a file that can be read by Weka [8] to
perform the classification.

This tool, as described, is to be publicly open for any
contribution, and hopefully makes a start point for an open-
source efficient tool to perform text classification for any
purpose.

III. RELATED WORK
Twitter data mining has been a hot topic of research in the last
few years. Nature of the data mined varies widely depending
on the aim and the final result expected. Consequently, the
techniques used to process data and extract the needed infor-
mation are different.

Akcora et al. [9] proposed a method to determine the
changes in public opinion over the time, and identify the news
that led to breakpoints in public opinion. In a related con-
text, Sriram et al. [10] proposed a method to classify tweets
depending on their natures into a set of classes including
private messages, opinions and event, etc.

However, most of the work has been focusing on the
content of the tweets and how to extract opinions of users
towards specific topics or objects. The work of Pang et al. [11]
presented the pioneer work for the use of machine learning to
classify texts based on their sentiment polarity. In their work,

20618 VOLUME 5, 2017

M. Bouazizi, T. Ohtsuki: Pattern-Based Approach for Multi-Class Sentiment Analysis in Twitter

the authors used unigrams, bigrams and adjectives in different
ways to classify a set of movie reviews into positive or nega-
tive. Other works iterated more on the idea, and new types of
features have been used for the classification, depending on
the aim and application: Boia et al. [12] and Manuel et al. [13]
proposed two approaches that, respectively, rely on emoti-
cons to detect the polarity of tweets and on slang words to
assign a sentiment score to online texts. These two works
proved how non-textual components can be used to detect the
polarity of a text.

More recent works went deeper, and new models have
been built: Gao and Sebastiani [14] proposed a recent
approach that focus in the repartition or the frequency of
sentiment classes in the set they analyze. Moving from
classification to quantification, the authors concluded that
using a quantification-specific algorithm presents a better fre-
quency estimation than using regular classification-oriented
algorithms.

Few works have been conducted on the multi-class sen-
timent analysis. Most of them focused on assessing the
sentiment strength into different sentiment strength levels
(e.g., ‘‘very negative’’, ‘‘negative’’, ‘‘neutral’’, ‘‘positive’’
and ‘‘very positive’’) or simply give numeric sentiment
scores to the texts [15], [16]. Nevertheless, other works were
conducted to classify texts into different sentiment classes:
Lin et al. [17], [18] proposed an approach that classifies doc-
uments into reader-emotion categories. They relied on what

they qualify as similarity features and word emotion features
along with other basic features. The approach, although it
shows some potential, is oriented towards the reader rather
than the writer. Therefore, the sentiment classes proposed are
different from what a writer might intend to show. Similarly,
Ye et al. [19] studied the problem of emotion detection of
news articles from reader’s perspective, and tried various
multi-label classification methods and different strategies for
features selection to conclude which are to be adopted to
solve the problem. Liang et al. [20] proposed an emoticon
recommendation system that recommends emoticons for
posted texts to help to author decide which emoticon to insert
to show what he intends.

IV. SENTA – A USER FRIENDLY TOOL FOR FEATURES
EXTRACTION FROM TEXTS
SENTA is a user-friendly tool we developed to extract differ-
ent features from the tweets, and texts in general, to perform
in a later step the classification of tweets/texts into different
classes. The features extracted vary widely, and cover the
context as well as the form of the text.

SENTA has several graphical interfaces that allow the user
to easily input his data, choose the features he wants to
extract, and save the output in different formats. In this work,
we have used SENTA to extract the necessary features that
we used to perform the task of multi-class sentiment analysis
in Twitter.

FIGURE 1. Pre-processing of tweets.

VOLUME 5, 2017 20619

M. Bouazizi, T. Ohtsuki: Pattern-Based Approach for Multi-Class Sentiment Analysis in Twitter

A. TOOLS
SENTA was built using Java and Java FXML. While many
libraries were used to build this program, mainly OpenNLP
was exploited in most of the tasks. OpenNLP has been used
to perform the NLP basic tasks such as the tokenization,
Part-of-Speech (PoS) tagging and the lemmatization of the
texts (i.e., tweets in our case).

B. CONVENTION
For the rest of this Section, the user of the program SENTA
will be referred to as ‘‘the user’’, while the Twitter user whose
tweet is processed will be referred to as ‘‘the twitterer’’

In addition, by interface, we mean a graphical user
interface of SENTA.

C. PRE-PROCESSING OF TWEETS
During this work, we pre-process each tweet as shown in
Fig. 1: we start by removing the URLs, tags at the beginning
of the tweets and irrelevant content. We then use OpenNLP
to tokenize the tweet, get the Part-of-Speech (PoS) tags of
the obtained tokens, and refer to both (tokens + PoS tags)
to get the lemmas of all the words. We then generate what
we call a negation vector of the tweet. A negation vector is
a vector having the same length as that of the tokens. If the
tweet contain a negation word (e.g., ‘‘not’’, ‘‘never’’, etc. ),

all the tokens (words) that come after, until the next punctua-
tion mark are considered as negated, and are attributed a value
equal to 1 in the corresponding negation vector. This will help
later detect which words are positive and which are negative.
Obviously, many works such as [21] present better solutions
to handle the presence of negation and polarity shifting in
sentiment analysis, however, we opted for this more straight-
forward, yet less complex and faster approach.

We also made an internal tool that decomposes the hashtags
into words referring to a dictionary of words occurrence
probability as we will describe later on in this work. This
decomposition is used also for detecting any sentiment hidden
in the hashtags. On a small set of hashtags (i.e., 100 different
hashtags) our tool reached a good accuracy of decomposition
that reached 88%.

D. GRAPHICAL USER INTERFACES
1) MAIN WINDOWS
a: PROJECT TYPE WINDOW
As mentioned above, SENTA was developed as a user-
friendly tool to extract different possible features from texts.
Therefore, to assist the user all over the process, different
interfaces are present.

From the first window shown in Fig. 2, the user chooses
whether he wants to open an already existing project,

FIGURE 2. The ‘‘Main’’ window of SENTA.

20620 VOLUME 5, 2017

M. Bouazizi, T. Ohtsuki: Pattern-Based Approach for Multi-Class Sentiment Analysis in Twitter

FIGURE 3. The ‘‘Open an existing project’’ window.

import features from an existing file (and eventually add them
to the ones he will extract once he goes to the next step), or
start a new project.

b: IMPORT PROJECT WINDOW
The import of an existing project supposes that a project
has already been created. SENTA allows the user to save an
existing project in a file with the extension ‘‘*.senta’’, along
with the different files required to load the project.

Fig. 3 shows the interface displayed when the user chooses
to open an existing project. He has the choice to browse his
computer to look for a project, or to select directly one of the
recently opened/created projects.

After the selection of the file, the user needs to click ‘‘Get’’
to collect the different options, parameters and features to be
collected:
• Project type: this refers to whether the sets used in the
existing project are a training set and a test set or a train-
ing set and a non-annotated set. The difference between
a test set and a non-annotated set will be explained later
in this section.

• Project name: the name of the project as saved earlier,
and this cannot be changed for the existing project, but
when saving the current project, the user might choose
a different name.

• Training and test files: these are the data sets used
previously.

• Sentiment classes: these are the classes that the tweets
are supposed to be classified to (extracted from the
training set)

• Features file: the different sets of features and fea-
ture parameters as selected previously for the opened
project.

• Extra files: these are used to make the feature extrac-
tion faster, if they have previously been extracted and
saved in the corresponding files. These will be explained
further later.

For the same project, the user can choose a different train-
ing and/or test set (or non-annotated set). He can also choose
not to use the old set of features, and select new ones.

c: IMPORT FEATURES WINDOW
As stated above, in addition to the extraction of features,
SENTA allows the import of extra features, which have been
extracted using external tools, so that they are added to the
set of features extracted by SENTA. Fig. 4 shows the window
where the extra features can be imported.

In addition to the training and the test/non-annotated sets
themselves, the user inputs 2 files corresponding to the extra
features.

VOLUME 5, 2017 20621

M. Bouazizi, T. Ohtsuki: Pattern-Based Approach for Multi-Class Sentiment Analysis in Twitter

FIGURE 4. The ‘‘Import features’’ window.

The user needs to specify the format of the file. Only a
Weka file (i.e., ‘‘*.arff’’), a text file (i.e., ‘‘*.txt’’ tabulation
separated) or a CSV file (i.e., ‘‘*.csv’’ comma separated), can
be imported.

The extra features extracted from both the training and
the test/non-annotated set need to be provided for all the
instances (tweets). In case one of the files is missing or in
case of inconsistency in terms of number of instances, the
extra features will be dismissed entirely.

Once the user specifies the location of all the files, he
needs to click on ‘‘Collect features’’ to get the tweets and
their features. The training and test/non-annotated sets have
a specific format required that will be discussed later on.
However, regarding the extra features files, they are highly
recommended to contain the Tweet ID field so that the fea-
tures can match the actual tweets collected from the data sets.
If such a field does not exist, the features will be attributed
automatically to the tweets in the same order. Obviously in
case of inconsistency (e.g., the number of lines in the data set
file and the features file are not equal) the features file will be
dismissed.

d: CREATION OF A NEW PROJECT WINDOW
However, during this work, no features, other than the ones
extracted with SENTA are used. Therefore, we opt for the

creation of a new project. To start a new project, the user is
supposed to provide two datasets: a training set and either a
test set or a non-annotated set as shown in Fig. 5. The training
set and test set have to contain at least the following attributes:
• Tweet ID: this is the unique ID of the tweet, that will be
used in the rest of the work to identify the tweet and that
will be used later to save the tweets features.

• Username: the name of the twitterer who posted the
tweet. While this information is not used for any purpose
during this work, this information might be needed in a
future extension (e.g., to detect the gender/location of
the user as extra features).

• Tweet message: the content of the tweet itself.
• Class: the user-defined class of the tweet.
The last attribute supposes that the tweets have already

been manually annotated by the user, and therefore can be
used for training and/or testing. For the same reason, if the
user decides to opt for a non-annotated set, in which case
he will extract the features and try to perform the prediction
of the classes of the different tweets, this attribute is not
supposed to be provided, and if given, it is simply ignored.

Once the files containing the data sets are selected, the user
can check the different classes by selecting ‘‘Load classes’’.
The user has also the possibility to add extra classes. While
this might seem irrelevant and meaningless at this point,

20622 VOLUME 5, 2017

M. Bouazizi, T. Ohtsuki: Pattern-Based Approach for Multi-Class Sentiment Analysis in Twitter

FIGURE 5. The ‘‘Start a new project’’ window.

these extra classes can be used later to extract extra features
(e.g., Unigram features), to enhance the accuracy of classifi-
cation. This will be discussed later on in this Section.

e: FEATURE SELECTION WINDOW
After the collection of the training tweets and the test/non-
annotated tweets, the user is supposed to select the features
he wants to extract. The features that can be extracted using
SENTA are divided into 7 different sets as shown in Fig. 6 that
we will cover later on. However, note that all the interfaces
that manage the extraction of features are similar.

The 7 sets of features are:
• Sentiment-related features
• Punctuation features
• Syntactic and stylistic features
• Semantic features
• Unigram Features
• Top words
• Pattern-related features
To select a set of features, the user has to check it, and then

customize it. The small question mark button next to the name
of the set of features opens a help window that explains what
the set of features does, and how to configure it.

The features selection along with their parameters can be
exported and re-imported for a future project any time.

Once the features and their associated parameters are set,
on the main window, the number of features to be extracted
for each family of features is displayed.

f: SAVE PROJECT WINDOW
The user is then called to choose the different options to save
his project as shown in Fig. 7, where he has to specify a
name for his project, a location for it to be saved, along with
the different save options including the type of output and
whether some extra data are to be saved or not.

Inside the project directory specified, a subfolder will be
created and named after the project name.

The features qualified as ‘‘Top words’’ and ‘‘Pattern-
related features’’ require the extraction of some words,
expressions or patterns from the training set (or an indepen-
dent set other than the test/non-annotated set) as we will
discuss later. However, given the fact that this procedure
takes some time, or that the user might prefer to extract
these dictionaries from an independent set, SENTA offers
the option to let the user import these from a different
source (and checks if they are valid or not). SENTA also
allows him to save the patterns and/or top words at this
stage that will be extracted from the current training set (this
requires that the user already selected these features to be
extracted).

VOLUME 5, 2017 20623

M. Bouazizi, T. Ohtsuki: Pattern-Based Approach for Multi-Class Sentiment Analysis in Twitter

FIGURE 6. The ‘‘Features selection’’ window.

The features, once extracted, can be saved in different
formats: a Weka file (i.e., ‘‘*.arff’’), a text file (i.e., ‘‘*.txt’’
tabulation separated) and/or a CSV file (i.e., ‘‘*.csv’’ comma
separated).

g: START EXTRACTION WINDOW
Once the project details have been set, the user can start the
feature extraction, and keep track of which task is currently
being run as well as the tasks already finished as shown in
Fig. 8. The time displayed is in seconds (s). The user can also
pause the task any time but this will not free any space in the
memory neither free the thread being run.

h: PROJECT SUMMARY WINDOW
The last interface in the main windows is a recapitulation of
the project along with the output files is displayed as shown
in Fig. 9.

The recapitulation includes in addition to the project name,
directory and type, the location and size of the training
and test sets, and the files generated along with the project
file.

From this point the user can go to the previous interface,
go back to the main interface or open in the system explorer
the project directory to browse the saved files.

2) FEATURE CUSTOMIZATION WINDOWS
Feature customization window appears when a user presses
the button ‘‘customize’’. For all the sets of features, we added
the button ‘‘Default’’ that selects by default the features that
we used to perform the multi-class classification in the rest of
this work to make it easy to replicate.

a: SENTIMENT FEATURES
Sentiment features are features which rely on the senti-
ment polarities of the different components of the text such
as the words themselves, emoticons, hashtags, etc. These
features are extracted using already-built dictionaries and
small sub-tools we use internally. Noticeably we referred
to SentiStrength to build our dictionary of emotional words,
however, we are currently building our own. Sentiment fea-
tures are divided into 5 sub-categories as shown in Fig. 10:

– Textual features: these are features that deal with the
textual component of the tweet. These include the following
features:
• Number of positive words
• Number of negative words
• Number of highly emotional positive words (i.e., words
having score returned by SentiStrength greater or equal
to 3)

20624 VOLUME 5, 2017

M. Bouazizi, T. Ohtsuki: Pattern-Based Approach for Multi-Class Sentiment Analysis in Twitter

FIGURE 7. The ‘‘Save project’’ window.

• Number of highly emotional negative words (i.e., words
having score returned by SentiStrength less or equal
to −3)

• Number of capitalized positive words
• Number of capitalized negative words
• Ratio of emotional words ρ(t) defined as

ρ(t) =
PW(t)−NW(t)
PW(t)+NW(t)

(1)

where t is the tweet, PW and NW are the total score of
positive words and that of negative words as returned
by SentiStrength. In case the tweet does not contain any
emotional word, ρ is set to 0.

– Emoticons-related features: these include the count of
positive, negative, neutral and joking (or ironic) emoticons.
Emoticons qualified of neutral are ones who do not show clear
emotion such as ‘‘(._.)’’ while joking emoticons are ones used
sometimes with ironical or sarcastic statements (e.g., ‘‘:P’’).
– Hashtags-related features: these include the count of

positive and negative hashtags. To decide on a hashtag’s
polarity, we defined a simple probabilistic model that decom-
poses the hashtag into words, and detects the polarity of the
resulting expression.

– Slang words-related features: these include the count of
positive and negative slang words. To extract these we refer to
a dictionary containing the most common slang words along
with their polarities.

– Contrast features: these detect whether there is any
contrast between the different components. By contrast we
mean the coexistence of a negative component and a positive
one within the same tweet, whether the two components have
the same nature (e.g., words, emoticons, etc.) or different
natures (e.g., words vs emoticons, etc.). In total 5 features are
extracted which include the contrast between words, between
hashtags, between words and hashtags, between words and
emoticons and between hashtags and emoticons.

b: PUNCTUATION FEATURES
Punctuation features are ones related to the use of punctuation
marks as well as the capitalization of words, etc. as shown in
Fig. 11. They are divided into 4 sub-categories:

– Punctuation marks: these include the number of full
stops, commas, semicolons, exclamation marks and question
marks.

– Parentheses and similar symbols: these include the
number of parentheses, brackets and braces.

VOLUME 5, 2017 20625

M. Bouazizi, T. Ohtsuki: Pattern-Based Approach for Multi-Class Sentiment Analysis in Twitter

FIGURE 8. The ‘‘Start of collection and project progress’’ window.

– Words and characters these include the count of words
and characters, the average number of words and characters
per sentence, etc.

– Apostrophe and quotation marks

c: SYNTACTIC AND STYLISTIC FEATURES
Syntactic and stylistic features are ones related to the use of
words and expressions in the tweet/text. They are divided into
3 sub-categories as shown in Fig. 12:

– Use of content words-related features: content
words are nouns, verbs, adjectives and adverbs. The fea-
tures extracted are the count and the ratio of each
aside.

– Syntactic features: these are related to the use of some
speech forms, proper nouns, and symbols.

– Use of words: these are features related to the use
of non-content words such as particles, interjections, pro-
nouns, negation. They also include the use of uncommon
words (which might obviously be content words). To judge
whether a word is common or not, we referred to a big
amount of texts collected online. We calculated the prob-
ability of use of the different words and qualified the top
5,000 words as ‘‘common’’ while the rest are considered as
‘‘uncommon’’.

d: SEMANTIC FEATURES
Semantic features are ones related to the meanings of words
in the language as well as the logic behind it. Fig. 13 shows
the features window. In the current version of the project, very
few features can be extracted. They include the use of opinion
words or expressions, the use of highly sentimental words,
the use of uncertainty words and the use of active and passive
forms.

e: UNIGRAM FEATURES
Unigram features are kind of special features that are
extracted with reference to dictionaries built according to the
user’s defined parameters. Since proposed by Pang et al. [11],
unigrams and n-grams in general, have been used as basic
features for sentiment analysis using machine learning. In the
different approaches, unigrams are collected from the training
data sets, and either the count or the presence of these uni-
grams are used as features for the classification. In this work,
we make use of WordNet [22] to collect unigrams related to
each sentiment class. The user is supposed to come up with
a small set of seed words few in number for each class, and
use WordNet to collect their synonyms and hyponyms down
to a certain depth. The choice of synonyms and hyponyms

20626 VOLUME 5, 2017

M. Bouazizi, T. Ohtsuki: Pattern-Based Approach for Multi-Class Sentiment Analysis in Twitter

FIGURE 9. The window displaying the ‘‘Summary of the project’’.

is based on the fact that these words are highly correlated
with the initial seed word, and usually describe the same
object, if not a more precise one. While synonyms refer
usually to equivalent terms, hypernyms and hyponyms show
the relationship between the more general term and its more
specific instances.

A hypernym, or a superordinate, is a broader term than a
hyponym, whereas a hyponym is a word or an expression
which is more specific than its hypernym. For example, for
the word ‘‘feeling’’, two of its direct hypernyms are ‘‘per-
ception’’ and ‘‘idea’’. The words ‘‘happiness’’, ‘‘anger’’ and
‘‘fear’’ are some of its hyponyms.
Hypernyms might lose some of the specificities of the

initial word, therefore, in our study, we collect only syn-
onyms and hyponyms of the seed words. On the other hand,
hyponyms also might lose the original meaning of the word,
and collide with some of other classes. Therefore, the depth
down to which we collect the hyponyms is set to a certain
value we refer to as Depth (or Dhypo, which is a parameter to
optimize by the user).

This is explained in Fig. 14 which shows how the dictio-
naries are extracted: we start with a set of seed words for each
sentiment class. We then collect the synonyms and hyponyms
to get to new sets of words, from which we further extract the

synonyms and hyponyms. The same process is repeated over
and over Dhypo times.

Fig. 15 show the different parameters set for unigram
features: in SENTA, the extracted words can be used as
individual binary features (i.e., a feature for each word that
detects whether or not that word appear in the tweet/text or
not) or they are all summed for each sentiment class, and the
count of words from each set on a given tweet is used as a
separate feature. They can also be separated based on their
PoS (i.e., nouns, verbs, adjectives and adverbs each aside) so
instead of having one group of words per sentiment class, the
user can get up to 4. This is because the number of words to
be extracted totally has to be set prior to the extraction. The
user can also choose to collect only words of just one or two
PoS out of the 4. This set of features has been proven to be
very efficient in detecting the sentiment of tweets as we will
discuss later in this paper.

The sets of seed words can be defined by pressing ‘‘manage
seedwords’’. By default, SENTA offers seed words for 12 dif-
ferent sentiment classes so that, if any of them is present,
when the user chooses to import default seed words, they are
added. The interface showing how to add a seed word is given
in Fig. 16. The user types the word, chooses its PoS and the
class it belongs to.

VOLUME 5, 2017 20627

M. Bouazizi, T. Ohtsuki: Pattern-Based Approach for Multi-Class Sentiment Analysis in Twitter

FIGURE 10. The ‘‘Sentiment features customization’’ window.

FIGURE 11. The ‘‘Punctuation features customization’’ window.

f: TOP WORDS
Top words, as their name indicate, are the words that occur the
most in the training set. Fig. 17 shows the parameters related

to this set of features: The user can choose the PoS of the
top words to be collected, whether he wants each PoS-related
words to be extracted separately, the number of Top Words

20628 VOLUME 5, 2017

M. Bouazizi, T. Ohtsuki: Pattern-Based Approach for Multi-Class Sentiment Analysis in Twitter

FIGURE 12. The ‘‘Stylistic and semantic features customization’’ window.

FIGURE 13. The ‘‘Semantic features customization’’ window.

per class or PoS, and again whether the features are binary or
numeric.

The two parameters ‘‘Min Ratio’’ and ‘‘Min Occurrence’’
define the criteria of extraction of top words. For a positive

sentiment class ‘‘A’’ (e.g. ‘‘Happiness’’), the ratio of occur-
rence of this word on the positive sentiment tweets over that
on all the negative sentiment tweets should be higher than
‘‘Min Ratio’’. In addition, it has to occur on the sentiment

VOLUME 5, 2017 20629

M. Bouazizi, T. Ohtsuki: Pattern-Based Approach for Multi-Class Sentiment Analysis in Twitter

FIGURE 14. Procedure of extraction of Unigrams using WordNet.

FIGURE 15. The ‘‘Unigram features customization’’ window.

class ‘‘A’’ more than the value set for the parameters ‘‘Min
Occurrence’’. In this work, when we run the multi-class
sentiment analysis on our training and test tweets, Top Words
have not been used as features, for the reason that they present
some redundancy with unigram features, since many of the
words on both collide.

g: PATTERN-RELATED FEATURES
The idea of our pattern-related features has been proposed in
our previous work [23], in which we proposed an approach

that relies on Part of Speech tags (PoS-tags) to extract
sarcastic patterns. In SENTA we elaborated more this kind
of features, and made a more generic approach to extract pat-
terns. Patterns are extracted based on the PoS-tags of words:
the different possible PoS-tags (36 in total, along with a 37th
one referring to the punctuation) are divided into different
groups, and given a sentence S, containing n different words,
the words of S are subject to different actions based on their
PoS-tag, and according to the rules defined by the user.

Fig. 18 shows the different parameters of the Pattern fea-
tures: initially, the user defines whether he wants his pattern

20630 VOLUME 5, 2017

M. Bouazizi, T. Ohtsuki: Pattern-Based Approach for Multi-Class Sentiment Analysis in Twitter

FIGURE 16. The ‘‘Seed words management’’ window.

to be used each as a separate feature, or summed based on
their length and sentiment class. If the features are separate
(i.e., each is a unique feature), only one pattern length is
taken into account, otherwise he can choose a minimal and
a maximal length for patterns. The user then chooses how
many categories he wants his features to be divided into,

and specifies the action to do for each category by pressing
‘‘Customize’’. The different actions for the different cate-
gories are given in Fig. 19: a word can be kept as it is,
lemmatized, replaced by a specific expression, or by a user
defined expression, etc.

The user is next supposed to specify for each PoS tag,
which category it belongs to by pressing the button ‘‘Define’’
which displays the window shown in Fig. 20.

Later on this work, when performing the multi-class classi-
fication, we will give a concrete example of how patterns are
extracted using SENTA. A pattern should occur on a given
sentiment class at least the value of the parameter ‘‘Min # of
Occurrences’’ times to be considered. Given a full pattern T
extracted from a tweet, and a pattern P extracted earlier
from the training set, we define the following resemblance
function [24]:

res(p, t) =






1, if the tweet vector contains the
pattern as it is, in the same order,

α, if all the words of the pattern
appear in the tweet in the correct
order but with other words in
between,

γ ·n/N, if n words out of the N words of
the pattern appear in the tweet in
the correct order,

0, if no word of the pattern appears
in the tweet.

FIGURE 17. The ‘‘Top words features customization’’ window.

VOLUME 5, 2017 20631

M. Bouazizi, T. Ohtsuki: Pattern-Based Approach for Multi-Class Sentiment Analysis in Twitter

FIGURE 18. The ‘‘Pattern-related features customization’’ window.

FIGURE 19. The different actions for different PoS-Tags categories.

If the patterns are used as unique features, each feature
takes the value of resemblance as defined. Otherwise, the
patterns are grouped into different groups based on their sen-
timent class and length as shown in TABLE 1 where L1 · · ·LM
are the different lengths of the patterns, and S1 · · ·SM are the
different sentiments (classes).

Given the K patterns extracted for the sentiment class Si
and the length Lj p the value of the feature Fij is

Fij =
K∑
k=1

res(pk, t) (2)

FIGURE 20. The ‘‘PoS-Tags categories customization’’ window.

TABLE 1. Pattern features.

Fij as defined measures the degree of resemblance of a tweet t
to patterns of class i and length j. Therefore, two more param-
eters are to be defined by the user which are α and γ .

20632 VOLUME 5, 2017

M. Bouazizi, T. Ohtsuki: Pattern-Based Approach for Multi-Class Sentiment Analysis in Twitter

E. EXTENSIBILITY
Currently, SENTA extracts some basic features that allow
performing tasks such as sentiment analysis, even for mul-
tiple classes. However, for more advanced tasks, we believe
that it requires more features to be added.

Currently, we are building some sets of features we quali-
fied as ‘‘advanced sentiment features’’, ‘‘advanced semantic
features’’ and ‘‘advanced pattern features’’ that extract deeper
features from the texts. However, other features related to
causality, conditionality, differentiation of informative and
interrogative form, etc. are to be added.

In addition, currently SENTA supports only English, which
presents a big limitation, since it makes it inapplicable for
other languages: we believe that making it support other
languages and/or detect automatically the language of the
text will add more value. Last, yet not the least, we plan to
implement some machine learning algorithms, or call Weka
internally to perform the classification, in case the user does
not want his features to be exported, rather prefers to make
the classification internally and get the results, so that he can
adjust the parameters while still running the program and
retry.

V. MULTI-CLASS SENTIMENT ANALYSIS – PROPOSED
APPROACH
A. PROBLEM STATEMENT
Given a set of tweets, we aim to classify each one of them to
one of the following 7 classes: ‘‘love’’, ‘‘happiness’’, ‘‘fun’’,
‘‘neutral’’, ‘‘hate’’, ‘‘sadness’’ and ‘‘anger’’. Therefore, from
each tweet, we extract different sets of features, refer to a
training set and use machine learning algorithms to perform
the classification.

We have chosen the aforementioned sentiment classes for
different reasons. First of all, given our observation during
our work [25], we mainly concluded that we needed a bal-
anced amount of data between negative and positive classes.
In addition, while the aforementioned sentiments are the ones
present the most in tweets as observed in [26].

B. DATA
For the sake of this work, we manually collected and prepared
2 datasets as follow:
• Set 1: this set contains 21 000 tweets which have been
manually classified into the 7 classes, each containing
3 000 tweets. This set is used for training. Therefore, in
the rest of this work, it will be referred to as the ‘‘training
set’’.

• Set 2: this set contains 19 740 tweets. All tweets are
manually checked and classified into the 7 classes. This
set will serve as a test set. Therefore, in the rest of this
work, it will be referred to as the ‘‘test set’’.

The structure of the dataset used is shown in TABLE 2.

C. FEATURES EXTRACTION
Under different emotional conditions, humans tend to behave
differently. This includes the way they talk and express their

TABLE 2. Structure of the dataset used.

feelings. Therefore, it might be important to rely, not only
on the vocabularies used, but also on the expressions and
sentence structures used under the different conditions, to
quantify and model these feelings. Therefore, in the rest of
this section, we rely on these assumptions to extract different
sets (or families) of features.

The features are extracted using SENTA, the tool we intro-
duced in Section IV.

1) SENTIMENT-BASED FEATURES
As stated above, sentiment-based features are ones based
on the sentiment polarity (i.e., ‘‘positive’’/‘‘negative’’) of the
different components of tweets. Out of the different features
offered by SENTA, we extract the following ones:
• The number of positive words and that of negative
words,

• The number of highly emotional positive words and that
of highly emotional negative words,

• The ratio of emotional words,
• The number of positive and negative emoticons,
• The number of positive and negative slang words.

2) PUNCTUATION-BASED FEATURES
While punctuation do not usually show any sentiments
explicitly, except for exclamation marks maybe, we believe
that the excessive use of some (e.g., question marks, excla-
mation marks, etc.) shows the strength of some sentiments.
For example, the following two tweets might show different
sentiments according to the annotators:

– ‘‘Why didn’t you go with him?’’
– ‘‘Why did you tell her???????’’
While in both examples, the twitterers are asking questions,

in the first one, the annotators agreed on classifying the
tweet as totally neutral, whereas in the second, some of them
pointed out that the twitterer is most likely angry or upset.
Even though, it is quite hard to tell whether it is the case or
not, we agree with the annotator on the fact that the second
tweet might be sentimental, regardless of what sentiment is
present, while the first one is neutral.

Out of the variety of punctuation features, after our prelim-
inary experiments, we decided to use the following ones:
• The number of full stop marks,
• The number of exclamation mark,
• The number of Question Marks,
• The total number of words,
• Number of quotation marks.

VOLUME 5, 2017 20633

M. Bouazizi, T. Ohtsuki: Pattern-Based Approach for Multi-Class Sentiment Analysis in Twitter

3) SYNTACTIC AND STYLISTIC FEATURES
In addition to the aforementioned sets of features, we also
extract features related to the use of words. We first extract
the ratios of nouns, verbs, adjectives and adverbs in the tweets
(out of all the words, including hashtags, symbols, etc.).
We also check whether or not the twitterer employed the
comparative and/or the superlative forms.

Furthermore, our experiments showed the usefulness of the
following features as good indicators of sentiment polarity, as
well as the sentiment class for some of them:
• The total number of particles,
• The total number of interjections
• The total number of pronouns, that of pronouns of
group I and II separately,

• The use of negation,
• The use, and the total number of uncommon words.

4) SEMANTIC FEATURES
Semantic features are features that focus on the meanings in
the language or the logic inside of the sentences. While these
features have not all been added, we used few of the existing
ones, including:
• The use of opinion words,
• The use of highly sentimental words,
• The use of uncertainty words,
• The use of the passive form of speech.

5) UNIGRAM FEATURES VS TOP WORDS FEATURES
‘‘Unigram features’’, as described above, are numeric fea-
tures that rely on WordNet to be extracted. In brief, a set of
seed words for each sentiment class is provided and we use
WordNet to enrich them. We then extract N features (where
N is the number of sentiments) by counting, for a given tweet,
how many words from each set exist in it.

‘‘Top words’’, on the other hands, are words that are
extracted from the training set itself. From all the training
tweets of a given sentiment S, we collect the most commonly
used words while making sure that the words extracted are
ones that show the given sentiment (i.e., that the number of
occurrences of any word in the tweets of the sentiment S
is higher enough than its occurrences in the tweets of the
other sentiments). These words are used later as indicators
(features) to detect the sentiment of a given tweet.

However, given the nature of these two sets of features,
a huge part of the words will overlap, and create a useless
redundancy that we do not need. Therefore, for the sake of
this work, we discarded ‘‘Top Words features’’, and focused
on what we qualified as ‘‘Unigram Features’’.

We started with 6 sets of words (i.e., for all the sentiments
except the sentiment ‘‘Neutral’’ containing in total 486 words,
with an average number of 81 words for each sentiment.
The initial set of words contains an overlapping equal to 0
between words of sentiments of opposite polarities, while
we tolerated some overlapping for sentiment of the same
polarities (e.g., the word ‘‘enjoy’’ is a seed word for both
sentiments ‘‘happiness’’ and ‘‘fun’’). The words selected can
be nouns, verbs, adjectives and adverbs.

Judging from the Fig. 21, the overlapping (or duplication)
of words in different sentiments including that in sentiments
of different polarities increases rapidly. Even though, these
words are being removed automatically, the duplication is a
crucial indicator of where to stop continuing collecting the
words. In this work, we were restricted to a depth equal
to 2.

As described above, we use the resulted sets of words to
extract 6 features, by counting the occurrences of the words
in the tweet to classify, taking into consideration the score of
the words.

FIGURE 21. Number of unigrams collected from WordNet using the seed words proposed.

20634 VOLUME 5, 2017

M. Bouazizi, T. Ohtsuki: Pattern-Based Approach for Multi-Class Sentiment Analysis in Twitter

6) PATTERN-BASED FEATURES
As described in Section IV, patterns are used as a comple-
mentary set of features to detect what unigrams cannot detect:
while in most of the cases, sentimental words are enough to
tell the sentiment of a sentence, in other cases, the person
employs some specific longer expressions to express his sen-
timent. For example, the following tweet shows sentiments of
happiness without employing any sentimental word showing
explicitly happiness:
‘‘You took me to the world I always dreamt of!!! Thank you

soooo much!’’
Even though the word ‘‘thank’’ refers to a positive attitude

or sentiment, the tweets contains sentiments of happiness that
the twitterer shows, and thanks her friend for.

To detect such expressions and learn them, we refer to
patterns of speech.

We basically divide the PoS tags into three categories: a
first one, referred to as EI, containing words which might
have emotional content, a second one, referred to as ‘‘CI’’,
containing non emotional words whose content is important
and a third one, referred to as ‘‘GFI’’, containing the words
whose grammatical function is important. If a word belongs
to the first category, it is replaced by the corresponding
expression shown in TABLE 3 along with its polarity (e.g.,
the word ‘‘good’’ would be replaced by POS-ADJECTIVE);
if it belongs to the second, it is lemmatized and replaced by
its lemma; and if it belongs to the third, it is replaced by the
corresponding expression shown in TABLE 3.

TABLE 3. Expressions used to replace the words of EI and GFI.

As mentioned above, the classification into categories is
done based on the PoS-tag of the word. The list of part-of-
speech tags and their category is given in TABLE 4.

TABLE 4. Part-of-speech tag categories.

We generate the vector of words for each tweet as
defined. For example, the following PoS-tagged tweet

‘‘He_PRP is_VBP dummy_JJ, _, why_WP would_VBD
you_PRP think_VBP I_PRP want_VBP to_TO go_VB
with_IN him_PRP !!!!_.’’ gives, among others, the following
pattern vector [PRONOUN VERB NEG-ADJECTIVE . why
VERB PRONOUN VERB PRONOUN POS-VERB to VERB
with PRONOUN .] that can be later used to generate smaller
patterns following the rules defined (i.e., minimal and maxi-
mal lengths of patterns).

In this work, we opted for the use of patterns of different
lengths, so that the features created are small in number to
make the classification task run faster.

Based on our previous work [25] and with few adjustments,
we set that the optimal values for Nocc, Lmin, Lmax, α and γ
as follows: 



Nocc = 3,
Lmin = 3,
Lmax = 10,
α = 0.1,
γ = 0.02,

On the other hand the parameter K has been introduced
in this work since we noticed a high imbalance between the
number of patterns for each class. Fig. 22 shows the classi-
fication accuracy using pattern-based features for different
values of K. According to the figure, the optimal value is 5.
Higher values enhance the accuracy during cross-validation,
but have no big impact on that of the test set.

In the next section, we evaluate the model we built, and
present the results of our experiments in the cases of binary,
ternary and multi-class classification.

VI. EXPERIMENTAL RESULTS
After the extraction of features, we run different test using
‘‘Random Forest’’ [27] classifier. We use 4 Key Perfor-
mance Indicators (KPIs) to evaluate the effectiveness of our
approach: Accuracy, Precision, Recall and F-measure:
• Accuracy refers to the overall correctness of classi-
fication. It measures the ratio of correctly classified
instances over the total number of instances.

• Precision refers to the fraction of the tweets correctly
classified, for a given sentiment, over the total number
of tweets classified as belonging to that sentiment.

• Recall refers to the fraction of tweets correctly clas-
sified, for a given sentiment, over the total number of
tweets actually belonging to that sentiment. In other
words, for one sentiment, this KPI is nothing different
from its accuracy.

• F-measure is defined as follows:

F-measure = 2 ·
precision · recall
precision+ recall

. (3)

A. BINARY CLASSIFICATION
We first run our experiment to detect the sentiment polarity
of tweets. For this sake, we remove the tweets belonging
to the class ‘‘Neutral’’, and grouped the other classes into

VOLUME 5, 2017 20635

M. Bouazizi, T. Ohtsuki: Pattern-Based Approach for Multi-Class Sentiment Analysis in Twitter

FIGURE 22. Accuracy of classification using pattern-based features for different value of K .

two main classes which are ‘‘Positive’’ and ‘‘Negative’’. The
former class contains tweets from the classes ‘‘Fun’’, ‘‘Happi-
ness’’ and ‘‘Love’’, while the latter contains tweets from the
classes ‘‘Hate’’, ‘‘Anger’’ and ‘‘Sadness’’. TABLE 5 shows
the results of classification. The accuracy obtained reaches
81.3%. Noticeably, the recall of negative tweets is the highest
(i.e., 83.5%), however the precision of positive tweets is the
highest (i.e., 82.0%). This means that tweets which are clas-
sified as positive are mostly positive. However, tweets which
have negative polarity tend to be classified more correctly as
shown in the confusion matrix presented in TABLE 6.

TABLE 5. Binary classification Accuracy, Precision, Recall and
F-measure.

TABLE 6. Binary classification confusion matrix.

The classification presents a noticeably low accuracy com-
pared with that of our previous work [25]. This is because in
our previous work, we exploited the information regarding
the detailed sentiment class for unigram features and pattern
features. In other words, when we extracted the features from
the training and the test set, we counted unigrams belonging
to the classes ‘‘Happiness’’, ‘‘Love’’, ‘‘Anger’’, etc. on tweets
of the training set and the test set. Furthermore, we extracted
patterns related to these detailed sentiments and used them

to measure the resemblance between the training and the test
tweets. While that was fair and acceptable given the fact that
we dispose of a training set with the detailed sentiment sub-
classes, for a more general case, where a person wants to
classify tweets into ‘‘Positive’’ and ‘‘Negative’’, such infor-
mation might not be provided, and so the training set will
contain tweets classified only as ‘‘Positive’’ and ‘‘Negative’’.
Therefore, in this work, we used the training set as a set of
tweets having initially only two classes: only two unigram
features are extracted, and patterns are also extracted from the
training set in only two subsets: positive patterns and negative
patterns.

B. TERNARY CLASSIFICATION
Despite its importance, binary classification supposes that
the given data are already known to be emotional. However,
Twitter contains many tweets which have no emotional polar-
ity such as news tweets, etc. Therefore, in this subsection we
add neutral tweets as shown before in the description of our
dataset. We then rely on the same set of features to classify
the tweets. As described previously, no information regarding
the sentiment sub-class is given or exploited here. The results
obtained are given in TABLE 7, and the confusion matrix of
classification is given in TABLE 8.

TABLE 7. Ternary classification Accuracy, Precision, Recall and F-measure.

The obtained results show that the introduction of the
third class decreases noticeably the accuracy to reach 70.1%.

20636 VOLUME 5, 2017

M. Bouazizi, T. Ohtsuki: Pattern-Based Approach for Multi-Class Sentiment Analysis in Twitter

TABLE 8. Ternary classification confusion matrix.

The new class (i.e., ‘‘Neutral’’) presents a low accuracy and
a low precision. This can be explained by the fact that the
amount of training data (i.e., number of tweets) for this class
is lower than that for the other classes. In addition, tweets,
regardless of their content tend to be polarized (i.e., either
classified as positive or as negative). This is because most
of the features used, except for the pattern features, are ones
that try to detect any sentimental component in a given tweet,
or find any resemblance of the tweet to ones in the training
set (which is highly unbalanced in favor of the sentimental
classes over the neutral class).

Overall, the results obtained are promising.

C. MULTI-CLASS CLASSIFICATION
In this subsection, we use the 7 sentiment classes that we
described in Section V. The classification results are given in
TABLE 9, while the confusion matrix is given in TABLE 10.

TABLE 9. Multi-class classification Accuracy, Precision, Recall and
F-measure.

TABLE 10. Multi-class classification confusion matrix.

Despite the number of classes, the accuracy obtained is
equal to 60.2%, with a precision that reaches 60.8%. More
interestingly, some sentiments seem to be easier to detect than
others. In particular, tweets belonging to the class ‘‘Love’’ and
those belonging to the class ‘‘Hate’’ were classified with an
accuracy equal to 75.2% and 90.9% respectively. This shows
that tweets belonging to these classes are easily distinguished
from other classes. This might be due to the fact that other
classes, such as ‘‘Happiness’’ and ‘‘Fun’’ for example are
very close to each other. Therefore, many tweets of one class
are classified as if they belong to the others.

The class ‘‘Neutral’’ on the other side, presents the low-
est precision. Many tweets, from all the other classes were
classified as neutral. While this does not go along with our
observations on [25]. We believe that the main difference is
that our current training set presents a cleaner reference for
training. The training set used in [25] contains a lot of noise,
and most of the noisy data are mainly neutral, but are used for
the other classes, which resulted in a misclassification of most
of the neutral tweets, and made the class ‘‘Neutral’’ present a
very low recall.

D. DISCUSSION
Classifying tweets is, to begin with, a difficult task given
the very limited size of tweets. The challenges presented
in Section II were tackled by many researchers, however,
remain still not completely solved. With reference to this
work, we can confirm that classifying tweets into separate
sentiment classes is a challenging task: as mentioned above,
many tweets present more than one sentiment. Therefore, a
more interesting task would be quantifying the sentiments
present in the tweet: a tweet should be attributed more than
one sentiment with different scores. The sentiments attributed
will represent all the existing sentiments detected in the tweet,
whereas the scores will represent the estimated weight of the
detected sentiment. We strongly believe that this would allow
to have a more accurate description of the sentiments in the
tweet, and solves the main issue that we encountered in this
work, which is the existence of multiple sentiments in the
tweet.

On a related context, even though we have ran several
experiments on our dataset, we cannot confirm that the values
set for the parameters defined are the optimal ones. SENTA
presents more than 12 different parameters, for the different
sets of features. We tried to optimize each set of parame-
ters, related to the same family of features aside, however,
this could be a non-optimal solution given the fact that the
machine learning algorithm used (i.e., Random Forest) does
not consider the features independently. It rather builds the
model with reference to all the features combined. However,
it is unpractical, and almost impossible to try all the combi-
nations of features to derive the optimal ones, that give the
highest accuracy.

Regarding the test set used itself, its manual annotation
was done on crowdflower.3 Several annotators from different
backgrounds participated in the annotation. To check the per-
formance of the annotators, we randomly picked 300 tweets,
annotated them, and compared the results with those done by
the random annotators. Interestingly, the sentiment polarity
(whether the tweet is positive, negative or neutral) of 91.3%
of the tweets was agreed on. However, when it came to
the detection of the sentiment itself, the rate of agreement
dropped to 72%. However, for many of the non-agreed on
tweets, we understood why the annotators decided to attribute
one sentiment over another, and this goes back to the issue

3https://www.crowdflower.com

VOLUME 5, 2017 20637

M. Bouazizi, T. Ohtsuki: Pattern-Based Approach for Multi-Class Sentiment Analysis in Twitter

we highlighted earlier: the existence of multiple sentiments
within the same tweet.

VII. CONCLUSION
In this paper, we have proposed a new approach for sentiment
analysis, where a set of tweets is to be classified into 7 dif-
ferent classes. The obtained results show some potential: the
accuracy obtained for multi-class sentiment analysis in the
data set used was 60.2%. However, we believe that a more
optimized training set would present better performances.

Throughout this work, we demonstrated that multi-class
sentiment analysis can achieve a high accuracy level, but
it remains a challenging task. A more interesting task is
to quantify sentiments present in the tweet. Therefore, in
a future work, we will use the results obtained for ternary
classification (which achieved an accuracy equal to 70.1%)
to classify tweets into ‘‘Positive’’, ‘‘Negative’’ and ‘‘Neutral’’.
The classified sentimental tweets (i.e., which have been clas-
sified as ‘‘Positive’’ or ‘‘Negative’’) will then be given scores
for the corresponding sentiment subclasses.

ACKNOWLEDGMENT
The research results have been achieved by ‘‘Cognitive
Security: A New Approach to Securing Future Large Scale
and Distributed Mobile Applications,’’ the Commissioned
Research of National Institute of Information and Commu-
nications Technology (NICT), JAPAN.

REFERENCES
[1] B. O’Connor, R. Balasubramanyan, B. Routledge, and N. Smith, ‘‘From

tweets to polls: Linking text sentiment to public opinion time series,’’
in Proc. Int. AAAI Conf. Weblogs Social Media, May 2010,
pp. 26–33.

[2] M. A. Cabanlit and K. J. Espinosa, ‘‘Optimizing N-Gram based text feature
selection in sentiment analysis for commercial products in Twitter through
polarity lexicons,’’ in Proc. 5th Int. Conf. Inform., Intell., Syst. Appl.,
Jul. 2014, pp. 94–97.

[3] U. R. Hodeghatta, ‘‘Sentiment analysis of Hollywood movies on Twitter,’’
in Proc. IEEE/ACM ASONAM, Aug. 2013, pp. 1401–1404.

[4] J. M. Soler, F. Cuartero, and M. Roblizo, ‘‘Twitter as a tool for predicting
elections results,’’ in Proc. IEEE/ACM Int. Conf. Adv. Social Netw. Anal.
Mining (ASONAM), Aug. 2012, pp. 1194–1200.

[5] A. Java, X. Song, T. Finin, and B. Tseng, ‘‘Why we Twitter: Understanding
microblogging usage and communities,’’ in Proc. 9th WebKDD 1st SNA-
KDD Workshop Web Mining Social Netw. Anal., Aug. 2007, pp. 56–65.

[6] K. Ghag and K. Shah, ‘‘Comparative analysis of the techniques for senti-
ment analysis,’’ in Proc. Int. Conf. Adv. Technol. Eng., Jan. 2013, pp. 1–7.

[7] Y. R. Tausczik and J. W. Pennebaker, ‘‘The psychological meaning of
words: LIWC and computerized text analysis methods,’’ J. Lang. Social
Psychol., vol. 29, no. 1, pp. 24–54, Dec. 2010.

[8] M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and
I. H. Witten, ‘‘The WEKA data mining software: An update,’’ SIGKDD
Explorations Newslett., vol. 11, no. 1, pp. 10–18, Jun. 2009.

[9] C. G. Akcora, M. A. Bayir, M. Demirbas, and H. Ferhatosmanoglu, ‘‘Iden-
tifying breakpoints in public opinion,’’ in Proc. 1st Workshop Social Media
Anal., Jul. 2010, pp. 62–66.

[10] B. Sriram, D. Fuhry, E. Demir, H. Ferhatosmanoglu, and M. Demirbas,
‘‘Short text classification in Twitter to improve information filtering,’’
in Proc. 33rd Int. ACM SIGIR Conf. Res. Develop. Inf. Retr., Jul. 2010,
pp. 841–842.

[11] B. Pang, L. Lillian, and V. Shivakumar, ‘‘Thumbs up?: Sentiment clas-
sification using machine learning techniques,’’ in Proc. ACL-02 Conf.
Empirical Methods Natural Lang. Process., vol. 10, pp. 79–86, Jul. 2002.

[12] M. Boia, B. Faltings, C.-C. Musat, and P. Pu, ‘‘A 🙂 Is worth a thousand
words: How people attach sentiment to emoticons and words in tweets,’’
in Proc. Int. Conf. Social Comput., Sep. 2013, pp. 345–350.

[13] K. Manuel, K. V. Indukuri, and P. R. Krishna, ‘‘Analyzing Internet slang
for sentiment mining,’’ in Proc. 2nd Vaagdevi Int. Conf. Inform. Technol.
Real World Problems, Dec. 2010, pp. 9–11.

[14] W. Gao and F. Sebastiani, ‘‘Tweet sentiment: From classification to quan-
tification,’’ in Proc. IEEE/ACM Int. Conf. Adv. Social Netw. Anal. Min-
ing (ASONAM), Aug. 2015, pp. 97–104.

[15] Y. H. P. P. Priyadarshana, K. I. H. Gunathunga, K. K. A. N. N. Perera,
L. Ranathunga, P. M. Karunaratne, and T. M. Thanthriwatta, ‘‘Sentiment
analysis: Measuring sentiment strength of call centre conversations,’’ in
Proc. IEEE ICECCT, Mar. 2015, pp. 1–9.

[16] R. Srivastava and M. P. S. Bhatia, ‘‘Quantifying modified opinion strength:
A fuzzy inference system for sentiment analysis,’’ in Proc. Int. Conf. Adv.
Comput., Commun. Informat., Aug. 2013, pp. 1512–1519.

[17] K. H.-Y. Lin, C. Yang, and H.-H. Chen, ‘‘What emotions do news articles
trigger in their readers?’’ in Proc. ACM SIGIR, Jul. 2007, pp. 733–734.

[18] K. H.-Y. Lin, C. Yang, and H.-H. Chen, ‘‘Emotion classification of online
news articles from the reader’s perspective,’’ in Proc. IEEE/WIC/ACM WI-
IAT, vol. 1. Dec. 2008, pp. 220–226.

[19] L. Ye, R. Xu, and J. Xu, ‘‘Emotion prediction of news articles from reader’s
perspective based on multi-label classification,’’ in Proc. Int. Conf. Mach.
Learn. Cybern., vol. 5. Jul. 2012, pp. 2019–2024.

[20] W. B. Liang, H. C. Wang, Y. A. Chu, and C. H. Wu, ‘‘Emoticon rec-
ommendation in microblog using affective trajectory model,’’ in Proc.
Asia–Pacific Signal Inf. Proc. Assoc. Ann. Summit Conf. (APSIPA),
Dec. 2014, pp. 1–5.

[21] R. Xia, F. Xu, C. Zong, Q. Li, Y. Qi, and T. Li, ‘‘Dual sentiment analysis:
Considering two sides of one review,’’ IEEE Trans. Knowl. Data Eng.,
vol. 27, no. 8, pp. 2120–2133, Aug. 2015.

[22] C. Fellbaum, WordNet: An Electronic Lexical Database. Cambridge, MA,
USA: MIT Press, 1998.

[23] M. Bouazizi and T. Ohtsuki, ‘‘Sarcasm detection in Twitter: ‘All your
products are incredibly amazing!!!’—Are they really?’’ in Proc. IEEE
Globecom, Dec. 2015, pp. 1–6.

[24] D. Davidov, O. Tsur, and A. Rappoport, ‘‘Semi-supervised recognition of
sarcastic sentences in Twitter and Amazon,’’ in Proc. 14th Conf. Comput.
Natural Lang. Learn., Jul. 2010, pp. 107–116.

[25] M. Bouazizi and T. Ohtsuki, ‘‘Sentiment analysis: From binary to multi-
class classification: A pattern-based approach for multi-class sentiment
analysis in Twitter,’’ in Proc. IEEE ICC, May 2016, pp. 1–6.

[26] M. Bouazizi and T. Ohtsuki, ‘‘Sentiment analysis in Twitter: From clas-
sification to quantification of sentiments within tweets,’’ in Proc. IEEE
GLOBECOM, May 2016, pp. 1–6.

[27] L. Breiman, ‘‘Random forest,’’ Mach. Learn., vol. 45, no. 1, pp. 5–32,
Jan. 2001.

MONDHER BOUAZIZI received the Bachelor
Engineering Diploma in communications from
SUPCOM, Carthage University, Tunisia, in 2010,
and the master’s degree from Keio University in
2017, where he is currently pursuing the Ph.D.
degree. He was a Telecommunication Engineer
(access network quality and optimization) for three
years with Ooredoo Tunisia.

20638 VOLUME 5, 2017

M. Bouazizi, T. Ohtsuki: Pattern-Based Approach for Multi-Class Sentiment Analysis in Twitter

TOMOAKI OHTSUKI (OTSUKI) (SM’01)
received the B.E., M.E., and Ph.D. degrees in
electrical engineering from Keio University, Yoko-
hama, Japan, in 1990, 1992, and 1994, respec-
tively. From 1994 to 1995, he was a Post-Doctoral
Fellow and a Visiting Researcher in electrical
engineering with Keio University. From 1993 to
1995, he was a Special Researcher of Fellowships
of the Japan Society for the Promotion of Sci-
ence for Japanese Junior Scientists. From 1998 to

1999, he was with the Department of Electrical Engineering and Computer
Sciences, University of California at Berkeley, Berkeley, CA, USA. From
1995 to 2005, he was with the Tokyo University of Science. In 2005, he
joined Keio University. He is currently a Professor with Keio University.
He has authored or co-authored over 140 journal papers and 340 inter-
national conference papers. He is involved in research on wireless com-
munications, optical communications, signal processing, and information
theory.

He is a fellow of the IEICE. He was a recipient of the 1997 Inoue Research
Award for Young Scientist, the 1997 Hiroshi Ando Memorial Young

Engineering Award, the Ericsson Young Scientist Award 2000, the 2002
Funai Information and Science Award for Young Scientist, the IEEE the
1st Asia-Pacific Young Researcher Award 2001, the 5th International Com-
munication Foundation Research Award, the 2011 IEEE SPCE Outstanding
Service Award, the 27th TELECOM System Technology Award, the ETRI
Journal’s 2012 Best Reviewer Award, and the 9th International Conference
on Communications and Networking in China 2014 (CHINACOM ’14) Best
Paper Award. He gave tutorials and the keynote speech at many international
conferences, including IEEE VTC, IEEE PIMRC, and so on. He was a
Vice President of the Communications Society of the IEICE. He served as
a Chair of the IEEE Communications Society, Signal Processing for Com-
munications and Electronics Technical Committee. He served a Technical
Editor of the IEEE Wireless Communications Magazine and an Editor of
Elsevier Physical Communications. He is currently serving an Area Editor
of the IEEE TRANSACTIONS ON VEHICULAR TECHNOLOGY and an Editor of the
IEEE COMMUNICATIONS SURVEYS AND TUTORIALS. He has served as general
co-chair and symposium co-chair of many conferences, including IEEE
GLOBECOM 2008, SPC, IEEE ICC2011, CTS, IEEE GCOM2012, SPC,
and IEEE SPAWC.

VOLUME 5, 2017 20639

Article 1 instructions

Topic: one page excluding reference for every posted article as below requirement

From the posted journal topic create the new topic(research title- A Hybrid Model for Emotion Detection and classification using Text data using NLP and Machine Learning) needs to have a gap in the literature that leads to the research question.

Must cover below three points

1) You need to discuss how you will use surveys of blog users to verify their perceptions of the emotions identified from a random sample of the texts used in the classification scheme.

2)State the research question for this research-

The research question: How to improve the machine learning classification scheme for the emotion classification from text data.

3) what is propelled? Or what is propelled type of Sentiment ?

The topic proposal is a general sketch of the dissertation – the topic, general reasoning behind the topic, as well as a potential thesis and thesis map for your Literature Review.  The topic proposal must clearly link to theory, identify what problem or gap in literature your proposed topic will address, and show a connection to program goals and core courses.

Requirement:

APA format

Grammarly check

No plagiarism