ABSTRACT
The rapid rise of Industry 4.0 is pushing manufacturing companies to rethink the way they work by implementing automation and digitalisation. The global COVID-19 pandemic has led to their immediate rather than transitory adoption. The digitisation of tools and working methods by using real-time data and information to make them available to operators on-site, as well as flexibility through the implementation of remote collaborative working solutions, are key factors for economic success.
Digital twins, together with extended reality technologies (comprising augmented and virtual reality), are becoming key tools to help factories plan production processes. This article presents how digital twin and XR technologies can be made accessible to operators to shape the future of maintenance.
Introduction
The current major priorities of chief executives to help preserve their businesses amid the Covid-19 pandemic are leading digital transformation projects (37%) and improving remote work experience (37%)[1]. While 52% of companies plan to cut or defer investments because of the pandemic, just 9% will make those cuts in digital transformation[2].
Industries affected by travel restrictions need to maintain the long-term sustainability and competitiveness of their sector. Despite the pandemic, global spending on digital transformation technologies and services is estimated to have grown by 10% in 2020[3]. The global market for digital twins was valued at $3.1bn in 2020 and is expected to reach $48.2bn by 2026[4]. Meanwhile, the extended reality (XR) market is expected to be valued at $31bn in 2021 and nearly $300bn by 2024[5].
The Centre for Digital Engineering and Manufacturing (CDEM) at Cranfield University
(www.cranfield.ac.uk/centres/centre-fordigital-engineering-and-manufacturing) hosts cutting-edge simulation and visualisation facilities and has access to high-end computing facilities for simulating the complex nature of the maintenance activities associated with complex engineering assets. It supports companies in their digital journey, ensuring the application of the latest technologies discovered in the research world. Among others, it collaborates with Babcock International, Rolls-Royce, BAE Systems, MoD, Siemens, and the Defence Science and Technology Laboratory (DSTL).
Digital twin technology
Performance, availability, reliability, and safety of assets can be improved through the implementation of digital twin technology. In 2020, Gartner named digital twins, under the concept of hyperautomation, as one of the ten technology trends of the year[6]. In 2021, consultancy firm Accenture named digital twins as one of the five technological trends that will shape the future[7]. Inevitably, digital twin technology will be part of every industry over the next decade, and the ones that do not follow this trend will be left behind. The size of the global market for digital twins is forecast to grow from $3.1bn in 2020 to $48.2bn in 2026, a compound annual growth rate of 58%, considering the post-Covid-19 scenario (see Figure 1). By 2026, the European digital twin market is expected to command over $9.5bn.
There are many definitions of a digital twin, but the core characteristic is that it is a data-centric technology. Generally speaking, a digital twin can be defined as a dynamic digital representation of a real entity.
It consists of three components: i) the physical space; ii) the virtual space; and iii) the connection layer between the two spaces. Creating the digital counterpart in the early stage of the mirrored asset’s life cycle would help establish and monitor the asset through life, supporting mostly maintenance, repair and overhaul. The application of this technology adds value to complex engineering assets – those assets that are composed of many components which may interact with each other. Such heavy engineered assets are characterised by a long life (around 30-40 years) and hence having a reliable support service infrastructure would significantly improve their through-life operational performance. Typical examples are ships, helicopters, planes, production systems, wind turbines, and power generation systems.
The main benefits of using digital twin technology are [8, 9, 10, 11]:
1. Risk & reliability: the product can be tested and validated in the digital world before it even exists
2. Improved process design testing: engineers can more effectively tailor and optimise production processes for the asset to be developed
3. Inter-department collaboration: having one unique digital platform containing the information allows collaboration across different departments, which leads to improved productivity and operational efficiency
4. Real-time monitoring and asset management: a digital twin can be accessible remotely, ensuring the in-time collection of data. This improves the management of the asset through life
5. Predictive analytics: big data coming from sensors in the real asset allow operators to actively analyse those data, feed them into artificial intelligence based algorithms, and forecast potential failures before they occur
6. Improving the decision-making process: asset-specific knowledge is included in the digital twin; the experience moves from humans to the digital twin, supporting operators in making efficient and improved decisions
7. Revenue growth opportunities: Reduced time to market for a new product and overall costs to produce a new product, and improvements in efficiency and cost of servicing the product.
From an executive report published by the consultancy Accenture in 2021, 87% of executives agree that digital twins are becoming essential to their organisation’s ability to collaborate in strategic ecosystem partnerships. The same report highlighted the importance of having a central intelligence hub to manage complexity, as 90% of executives believe their organisation requires a mission control, or central intelligence hub, to gain insights into complexities and model the organisation’s processes, people, and assets [6].
Building the digital twin allows users to build a risk-free “playground” to explore new product ideas and limitless “what-if” scenarios. The difference between computer simulations and a digital twin lie in the data. A digital twin uses data from the connected sensors to represent the “as-is” condition. By incorporating this data into the virtual model, performance can be tested in a real-world scenario. In an example from Siemens [12], engineers developed a digital twin of the product to test and simulate future features of the product; a digital twin of its production, optimising the production in advance, saving time and costs; and a digital twin of its performance during operation, to perform predictive maintenance, preventing downtime, and optimising energy consumption.
Today at Cranfield the CDEM is developing, among others, digital twins to monitor the through-life degradation of complex engineering assets. This includes the employment of accurate degradation models to feed artificial intelligence algorithms, which eventually lead to a predictive maintenance approach. In this way, we can move the paradigm from reactive maintenance to proactive maintenance, avoiding downtime and optimising maintenance planning at optimal costs.
In our opinion, this is not just an opportunity for big enterprises. Digital twins could also be an asset for SMEs. Digital technologies are becoming more and more agile, creating a whole new ecosystem providing both competitive and sustained advantage.
Augmented Reality (AR) capabilities and impact on maintenance AR is an emerging technology that has grown immensely in recent years in both the hardware and its applicability to improve real-time maintenance. New head-mounted devices, such as the HoloLens 2 which supports full hand-tracking and a wide field of view, have revolutionised the use of AR solutions in modern maintenance. The HoloLens has become an attractive interactive solution for assisting maintainers’ work via training or visual guidance. A few benefits of ARenabled maintenance that have been proved are reduced human error, reduced downtime and reduced execution time while increasing productivity and compliance [13].
Case studies have shown how AR solutions can be beneficial in reducing error rates as well as improving efficiency. GE Aviation recorded efficiency improvement of up to 12 % with its AR case study[14]. Boeing was able to reduce error rates as well as decrease its wiring production time by 25%[15].
AR is mature enough to change the way companies approach industrial tasks and is one of the key tools for empowering operators and ensuring effective human-machine interaction. AR headsets provide the user with an array of options to visualise data. For instance, thermography inspection data can be projected on to the headset together with visual information and any augmented data that the user might use to make a maintenance decision on the inspection that has just been completed.
The following are a selective few applications of AR in the maintenance and inspection sector:
• A Canadian start-up, Longan Vision [16], has developed a thermal vision AR system that enhances firefighters’ vision. It has the potential to prevent casualties by allowing firefighters to see through smoke.
• The US Army [17] has been supplied with 120,000 Microsoft HoloLens headsets in a contract worth $21.88bn. This makes intelligent heads-up displays and augmented training environments possible. The use of XR technology in a military environment is a clear display of confidence.
• GE Renewable Energy has taken advantage of AR headsets for its wind turbine technicians, to provide information for completing wiring insertions for wind turbines while still allowing hands-free work. This approach has been proved to decrease assembly time by 34% [18].
• Netherlands Aerospace Centre NLR has incorporated augmented reality to provide immersive and collaborative training [19]. Its training systems can give students greater insight into how aircraft systems function and allow students to work together. By connecting multiple HoloLens headsets, it can create and share the same 3D model with students, as if they are all standing in front of an aircraft.
AR does more than show typical components. It is even able to simulate situations and mechanisms that are not possible when an aircraft is on the ground or are not accessible by traditional training methods. An example would be the landing gear of an aircraft or being able to provide repair simulations with no risk of causing damage to the aircraft. This provides much more variety in training, and researchers found AR more involving and exciting compared with traditional methods. NLR stated that it also shortened the total length of training time required and eased the process to real-life maintenance [20].
Not every company has programming expertise, which can be costly. The AR solutions for industry company Diota [22] provides a turnkey digital solution that helps maintenance technicians manage increasingly complex products and processes. It improves the quality of maintenance by reducing the risk of error by up to 90% and the time it takes to complete tasks by up to 80%. Moreover, it provides better information traceability and operational feedback.
Remote working with Virtual Reality
Teleoperation in maintenance and inspection is used to remotely control a robot to perform complex and potentially hazardous tasks for humans, for example in areas exposed to high levels of ionising radiation when maintaining nuclear facilities. It offers the possibility of improving the physical and cognitive abilities of the operator by increasing strength and endurance, for example when the task requires heavy lifting. Remotely operated maintenance robots are used for inspection, environmental surveys and telemanipulation, which may include grasping, cutting, welding and screwing tasks.
Various robotic platforms exist for maintenance and inspection. These can be surface robots equipped with suction devices, mobile ground robots with wheels or continuous tracks, or aerial robots. EasyJet [23] began in 2015 to test the use of drones to scan and assess aircraft, providing a damage report for subsequent maintenance work. Singapore-based aerospace, electronics, land systems and marine technology specialist ST Engineering [24] includes robots in its smart maintenance, repair and overhaul vision to carry visual inspections of aircraft.
In 2018, Rolls-Royce demonstrated the future of engine maintenance with miniature robots that can crawl inside engines [25], aiming to speed up inspection processes or removing the need to take an engine off an aircraft during maintenance work. In the energy sector, Sensabot [26], a remotely operated mobile inspection robot, is used by Shell to inspect oil and gas field equipment, and go wherever humans can, but without needing extra protection.
Most industries make workplace safety a top priority. The need for an effective interface is essential to communicate the robot’s position in real time, monitor status and plan work strategies to enhance productivity. Existing teleoperation approaches typically rely on 2D monitors, and display information such as visual information on where the robot is operating, together with the sensor data. However, the sense of presence remains limited, resulting in a loss of efficiency. When performing remote maintenance, immersive visual perception of the remote physical environment is important for the success rate of the task and the speed of execution. XR has been identified as a promising technology for providing teleoperation interfaces for real-time control of physical robots and boosting the turnaround time for remote maintenance activities.
The objectives of using XR as a human-robot interface are to enable more natural and intuitive control, to make it accessible to a wider population of users. Improving the efficiency of the interface leads to a reduction in operator error, keeping the operator focused on the task. At Cranfield University, the concept of telexistence, the fusion of robotics and VR which gives a sense of existence in the remote robot environment, is being explored. The aim is to provide an effective way of working at a distance for maintenance and inspection activities with a sense of presence and effective data integration for the operator.
Until now, only a few companies have investigated the capabilities of VR-based teleoperation. For instance, in 2020, the Japanese start-up Telexistence announced the development of “Model-T” [27], a robot using VR, telecommunication, the cloud, and haptics to perform remote manipulation. The robot is mounted with 22 DOF joints. Communication between the robot’s camera and the operator’s screen has a latency of only 50 milliseconds. All data is transmitted via the Microsoft Azure platform. The robot is equipped with specialised robotic hands for gripping and has already been successfully tested in several shops in Japan to put items on the shelf.
The start-up Extend Robotics in the UK is developing a cloud-based platform to remotely control a robotic arm using an intuitive realtime VR interface. Combined with volumetric video of the remote physical environment, it improves telepresence. The aim is to enable the operator to do his or her job better and faster while using an affordable solution. Recent experiences with Fine Energy, a company operating in the renewable energy sector, have put Extend Robotics’ concept into practice for remote inspection and maintenance of wind turbine electrical panels [28].
Another approach to remote maintenance is being developed by Sarcos Defense, a division of US-based Sarcos Robotics. It is working on industrial powered exoskeletons to meet maintenance and logistics needs. In late 2020 it partnered with the US Navy for the Guardian DX [29] variant, a robotic platform with two remote-controlled robotic arms that incorporates AI-based adaptive control to reduce the operator’s cognitive load while performing tasks. It is designed to be used on existing mobile access platforms for operations such as structural non-destructive testing, cutting and welding at height, and lifting and handling of heavy components, while keeping the operator at a safe distance.
One of the main benefits of using an XR headset to operate a robot for maintenance and inspection is portability. There are many barriers to the maintenance of remotely located assets, especially during the Covid-19 pandemic: physical health, location, and security clearance. Maintaining the availability of the workforce and expert users helps build a resilient factory that recovers from failures. Since the launch of commercial VR headsets five years ago, XR has matured but is not yet widely implemented. The exploration of XR as a viable interface solution for teleoperation is still in the initial stages.
The wider adoption of XR technologies by companies as a tool for performing remote maintenance tasks depends on a few parameters. Photorealistic graphics, resolution, high frame rates, a wider field of view and stand-alone devices are attributes shared with the entertainment industry. Manufacturing needs XR to be integrated into product lifecycle management to provide high fidelity and access to the digital twin for immersive visualisation. This would make XR an integral part of production flow and make possible a novel approach to maintenance and remote inspection.
The XR market is still young and there is still some way to go before the technology becomes mainstream in the maintenance and inspection sectors, with many challenges remaining to major adoption. Graphics processing IT specialist Nvidia is working on the future of XR streaming on Microsoft Azure. Nvidia Cloud XR [30], which will be available later in 2021, will allow high-fidelity experiences from the cloud by removing the need for high-end graphics and computing power, making XR available worldwide on a lightweight all-in-one device. Lastly, another emerging technology that will surely benefit XR and robotics is the use of blockchain for cyber security. Blockchain can be used in any sector to create a standard security protocol superior to end-to-end encryption with optimum security and privacy of data.
Conclusion
Emerging technologies offer the opportunity to implement new strategies for collaboration between business units. They enable rapid communication and problem-solving from the design phase to the operation site and the shop floor. The integration of XR and digital twin technologies into production flow will be a key aspect of their major adoption by industries. Accenture, in its 2021 technology trends report, stated that 65% of executives expect their organisation’s investment in intelligent digital twins to increase over the next three years. With the rapid growth of digital twins and complementary emerging technology such as XR, we can expect to see multidimensional digital assets that enable engineers to mirror industry assets to a precise degree. This would open the doors to intelligent and immersive digital twins with advanced visualisation, easily pushing the virtual asset to any user, from office workers to remote and on-site maintainers, in an interactive personalised manner while being able to evolve with the industry environment. It would allow the industry to take advantage of these developments, enhancing and working in a complementary way instead of using isolated solutions.
Machine learning will empower robots by giving them the ability to learn from and collaborate with humans, to speed up tasks by taking data into account. Collecting data from the machine will predict the best time to perform maintenance. Predictive maintenance will help companies avoid system failures that stop production. The exploitation of sensor data is a key aspect of the future of maintenance.
This level of flexibility can be achieved by implementing emerging digital technologies to support the future of remote maintenance. Making systems more sustainable through automation and monitoring, tomorrow’s maintenance will be efficient and optimised towards zero downtime.
To learn more about how Cranfield University can help your organisation to build a resilient and agile digital twin, telexistence and augmented reality capabilities for maintenance, contact:
Davide D’Amico (r.damico@cranfield.ac.uk)
Prof. John Ahmet Erkoyuncu
(j.a.erkoyuncu@cranfield.ac.uk)
Note: The content presented in this article is based on publicly available information and data collected and represented through fundamental research. While every effort has been made to ensure the accuracy of the information contained in this article, the authors cannot accept responsibility for errors. The authors reserve the right to make changes to this article.
References
[1] Statistica, “CEOs’ top priorities for CIOs to preserve business amid COVID-19 2020,” November 2020. [Online]. Available: https://www. statista.com/statistics/1131695/top-3-priorities-forcios/.
[2] PwC, “2020 Global Digital IQ,” 2020. [Online]. Available: https://www.pwc.com/us/en/library/digital-iq.html.
[3] IDC, “IDC Spending Guide,” May 2020. [Online]. Available: https://www.idc.com/getdoc.jsp?containerId=prUS46377220.
[4] MarketsandMarkets, “Digital Twin Market – Global Forecast to 2026,” September 2020. [Online]. Available: https://www.marketsandmarkets.com/Market-Reports/digital-twin-market-225269522.html.
[5] Statista, “Extended reality (XR): AR, VR, and MR – statistics & facts,” March 2021. [Online]. Available:
https://www.statista.com/topics/6072/extendedreality-xr/.
[6] Accenture, “Technology Trends 2021,” 2021. [Online]. Available: https://www.accenture.com/gben/ insights/technology/technology-trends-2021.
[7] Gartner, “Gartner Top 10 Strategic Technology Trends for 2020,” October 2019. [Online]. Available: https://www.gartner.com/smarterwithgartner/gartner-top-10-strategictechnology-trends-for-2020/.
[8] Capgemini, “Digital twin within the supply chain – the benefits,” Capgemini, 30 03 2021. [Online]. Available: https://www.capgemini.com/se-en/2021/03/digital-twin-within-the-supplychain-the-benefits/. [Accessed 27 04 2021].
[9] M. M. Armstrong, “Cheat sheet: What is Digital Twin?,” IBM, 04 12 2020. [Online]. Available: https://www.ibm.com/blogs/internet-of-things/iotcheat-sheet-digital-twin/. [Accessed 27 04 2021].
[10] W. Khan, “How can terminal operators benefit from digital twin technology amid pandemic uncertainty?,” Siemens, 05 2020. [Online]. Available: https://new.siemens.com/mea/en/company/stories/energy/terminal-operatorsbenefit-from-digital-twin.html. [Accessed 27 04 2021].
[11] A. Parrott and L. Warshaw, “Industry 4.0 and the digital twin,” Deloitte, 2017. [Online]. Available: https://www2.deloitte.com/content/dam/Deloitte/cn/Documents/cip/deloitte-cn-cip-industry-4-0-digital-twin-technology-en-171215.pdf. [Accessed 27 04 2021].
[12] Siemens, “Digitalization in industry: Twins with potential,” [Online]. Available: https://new.siemens.com/global/en/company/stories/industry/the-digital-twin.html .
[13] M. A. a. M. Annunziata, “Augmented Reality Is,” March 2017. [Online]. Available: http://donar.messe.de/exhibitor/hannovermesse/2017/A136112/harvard-business-review-articleeng-512156.pdf.
[14] G. Aviation, “GE Aviation Pilots AR to Cut Errors, Save Millions,” July 2017. [Online]. Available: https://upskill.io/landing/ge-aviation-case-study/.
[15] UpSkill, “Boeing works with Upskill to Cut Production Time by 25%,” April 2017. [Online]. Available: https://www.youtube.com/watch?v=nUn9VvzzjdY&ab_channel=Upskill.
[16] Longan Vision, “FUSION VISION SYSTEM,” [Online]. Available: https://www.longanvision.com/fvs.
[17] Microsoft Blog, “Army moves Microsoft HoloLensbased headset from prototyping to production phase,” 2021. [Online]. Available: https://blogs.microsoft.com/blog/2021/03/31/armymoves-microsoft-hololens-based-headset-fromprototyping-to-production-phase/.
[18] UpSkill, “Upskill & GE Renewable Energy: Wind Turbine Wiring Productivity Study Using Skylight,” March 2017. [Online]. Available: https://www.youtube.com/watch?v=n5LhQqggGTE&ab_channel=Upskill.
[19] Royal NLR, “Augmented and Virtual Reality,” [Online]. Available: https://www.nlr.org/capabilities/augmented-and-virtual-reality/.
[20] A. Nabben, “Augmented Reality taking off in MRO training,” May 2017. [Online]. Available: https://www.nlr.org/article/augmented-reality-taking-offmro-training/.
[21] PTC, “Vuforia Chalk: Remote Assistance Powered by Augmented Reality,” 2020. [Online]. Available: https://www.ptc.com/en/products/vuforia/vuforiachalk.
[22] Diota, “Digital-assisted Operator,” [Online]. Available: https://diota.com/en/digital-assistedoperator.
[23] IDG.tv, “EasyJet testing aircraft inspection by drone,” June 2015. [Online]. Available: https://www.youtube.com/watch?v=6vJtyjN8U_M.
[24] ST Engineering, “Smart MRO,” [Online]. Available: https://www.stengg.com/smart-mro.
[25] Rolls-Royce, “Rolls-Royce demonstrates the future of engine maintenance with robots that can crawl inside engines,” July 2018. [Online]. Available: https://www.rolls-royce.com/media/pressreleases/2018/17-07-2018-rr-demonstrates-thefuture-%20of-engine-maintenance-with-robots.aspx.
[26] Shell, “A BIONIC INSPECTOR ROLLS IN,” [Online]. Available: https://www.shelnside-energy/a-bionicinspector-rolls-in.html.
[27] Telexistence Inc., “Telexistence Model T at Lawson Takeshiba,” October 2020. [Online]. Available: https://www.youtube.com/watch?v=WLDucRUwJbo.
[28] Extend Robotics, “Remote Inspection and Maintenance of Electrical Panel for Wind Turbine,” April 2021. [Online]. Available: https://www.youtube.com/watch?v=WpbtYGRSxgU.
[29] Sarcos Defense, “Teleoperated Dexterous Robot for Defense,” 2020. [Online]. Available: https://www.sarcos.com/defense/guardian-dx/.
[30] Nvidia, “NVIDIA Delivers High-Quality XR Streaming on Microsoft Azure,” April 2021. [Online]. Available: https://developer.nvidia.com/blog/nvidia-delivers-high-quality-xr-streaming-onmicrosoft-azure/.
[31] K. Panetta, “5 Trends Emerge in the Gartner Hype Cycle for Emerging Technologies, 2018,” Gartner, 16 08 2018. [Online]. Available: https://www.gartner.com/smarterwithgartner/5-trendsemerge-in-gartner-hype-cycle-for-emergingtechnologies-2018/. [Accessed 05 2021].
[32] K. Panetta, “5 Trends Drive the Gartner Hype Cycle for Emerging Technologies, 2020,” Gartner, 18 08 2020. [Online]. Available: https://www.gartner.com/smarterwithgartner/5-trendsdrive-the-gartner-hype-cycle-for-emergingtechnologies-2020/. [Accessed 05 2021].
Davide D’Amico
PhD Researcher (Digital Twin)
Usama Shazlee
PhD Researcher (Augmented Reality)
Damien Mazeas
PhD Researcher (Telexistence)
Sri Addepalli
Lecturer in Degradation Assessment
Samuel Court
Research Fellow in Augmented Reality and Laboratory Manager
John Ahmet Erkoyuncu
Professor of Digital Engineering, head of the Centre of Digital Engineering and Manufacturing. Cranfield University.