Drones in Libraries : The Development of an Interdisciplinary Research Service Using Drones and 3 D Modeling Technologies at Ryerson University Library

At Ryerson University Library, we collaborate with researchers across a variety of disciplines that require capturing and modeling the real world in 3 dimensions (3D). To do so, several 3D reconstruction technologies have been applied, varying in cost and ease of use. Some applications require capturing large areas for which a Remotely Piloted Aircraft System (RPAS) or “drone” presents a cost effective option for data acquisition.


Introduction
At Ryerson University Library, we collaborate with researchers across a variety of disciplines that require capturing and modeling the real world in 3 dimensions (3D). To do so, several 3D reconstruction technologies have been applied, varying in cost and ease of use. Some applications require capturing large areas for which a Remotely Piloted Aircraft System (RPAS) or "drone" presents a cost effective option for data acquisition.
On June 1, 2019, new rules for flying a RPAS in Canada came into effect, requiring drone pilot certification to operate any drone between 250 g and 25 kg. In response to new regulations and the needs of our researchers, the Library has initiated the development of a research service dedicated to supporting the use of drones and 3D modelling technologies. This service is well positioned in a central research hub that minimizes researchers' need to acquire expertise and licensing given the new RPAS regulations. In addition to data collection via flights, the library provides consultations supporting 3D reconstruction technologies.
Before cancellation due to the Covid-19 pandemic, the joint CAG/CCA/CARTO-ACMLA conference -CAG 2020: Resilience on a Dynamic Planet -provided a national venue to showcase our progress to date. A 1.5 hour workshop was to be conducted both as a conference presentation and live demonstration of drone technology. The agenda included an introduction to drone licensing in Canada, a showcase of our missions and resulting deliverables to date, and a demonstration of the imagery acquisition process through conducting a live autonomous flight. Another goal was to connect with like-minded researchers in the field and explore the potential for future collaboration. This report will summarize our workshop content and outline existing collaborations and future directions for our research and service.

Regulations and Legal Requirements for flying RPAS in Canada
On June 1, 2019, drone pilot certification became a requirement to legally operate RPAS between 250 g and 25 kg. Along with acquiring the pilot certificate, the pilot must register the RPAS with Transport Canada and mark the drone with the registration number. A drone that is less than 250g does not require registration or a drone pilot certificate. (Flying your drone safely and legally, 2020). A complete list of regulations and requirements is outlined by Transport Canada on their Drone Safety webpage.

Category of Drone Operation
Two categories for drone operation exist in Canada which follow specific rules related to any flight. These main categories are Basic and Advanced operations which both require that the operator and/or crew maintain a Visual-Line-of-Sight (VLOS) with the RPAS at all times.
Basic Operations: Three main conditions exist for operating a drone with a Basic drone pilot certificate. The operator must conduct flight in uncontrolled airspace. Transport Canada defines uncontrolled airspace as areas with no air traffic control, at least 1.8 km away from a heliport and 5.6 km away from airports. The drone must be flown a minimum of 30 metres (100 feet) horizontally from bystanders and may never be flown over bystanders. To acquire the pilot certificate for Basic Operations, the operator must pass an online examination and may be asked to show proof that the related knowledge is retained over time (Find your category of drone operation, 2020).

Advanced Operations:
If any of the conditions outlined under Basic operations are not met, the operator must acquire a pilot certificate for Advanced Operations. Under Advanced operations, a drone pilot may request air space approval from NAV CANADA to conduct a flight in controlled airspace, may fly within 30 metres (100 feet) horizontally from bystanders, and may apply for a Special Flight Operations Certificate (SFOC) to fly over bystanders. To acquire the pilot certificate for Advanced Operations, two knowledge requirements exist. The operator must first pass an online examination and then schedule an in person flight review with a reviewer that is associated with a drone flight school (Find your category of drone operation, 2020).

Development of an Interdisciplinary Research Service
Our research consultation service related to drones has grown based on a partnership initiated between the Ryerson University Library Collaboratory and the Geospatial Map and Data Centre (GMDC). The Ryerson University Library Collaboratory is an interdisciplinary research hub that provides Ryerson faculty and graduate students with physical space, technology and access to consultation services to facilitate research. The GMDC houses and provides access to a geospatial data inventory, hardware, software and consultation services supporting the effective uses of GIS and related technologies. Although our geospatial inventory is extensive, specific demand for imagery and digital elevation data including point cloud data in the .las file format has been steadily growing over time. This demand, the changes to regulations for flying drones in Canada, and complementary knowledge bases ignited a collaboration between Jimmy Tran (Research Technology Officer in the Collaboratory) and Dan Jakubek (GIS and Map Librarian) to develop this service.
Since acquiring pilot certificates for Advanced Operations in June 2019, the Library has worked to establish a research community at Ryerson University related to this technology. Outreach with faculty from related disciplines led to a preliminary event in the Library Collaboratory to help guide the scope of our service and to stimulate collaboration. As a result, we have identified various needs of our research community ranging from guidance related to the drone licensing process, assistance with flight planning and imagery capture in diverse environments, processing and creation of data products with specialized software, and support related to visualization and analysis. A selection of these collaborations will be highlighted in this article.

Overview of Library Collaboratory Drones and Technology
Over the past year, the Library has experimented with various drones in the imagery acquisition process. To date we have worked exclusively with DJI drones (the current industry leader in drone development). The majority of our flights have been conducted with the DJI Mavic Pro. Although this drone is the original design in the Mavic Series, it has proven to be a reliable and cost effective solution in both manual and autonomous flights. The DJI Spark (now out of production) is a smaller drone which was acquired to be flown in dense/urban environments which the Mavic Pro could not easily navigate. In response to the new Canadian regulations requiring a license to fly drones in Canada weighing above 250 g, DJI replaced the DJI Spark with the Mavic Mini (this drone features similar specifications when compared to the Spark and a total weight of 249 g with memory card). Given its weight, the Mavic Mini does not require a drone pilot certificate to be flown in Canada and provides an excellent option for flight training or activity in controlled airspace where requests for airspace approval are not possible.
The DJI Mavic Pro, Spark, and Mavic Mini all provide the ability to acquire imagery given their built-in cameras on gimbals; however, they do not have the capacity to handle significant payloads required to mount specialized cameras and/or sensors. Based on consultations with our research community and data requests via the GMDC, datasets acquired via LiDAR (Light Detection and Ranging) technology e.g. 3D Point Cloud and Digital Elevation Models (DEMs) are a priority. LiDAR is a laser based technology that produces 3D representations of terrestrial or airborne features (LIDAR 101, 2020) and requires an enterprise drone solution given the payload required for data acquisition. In response to this demand, the Collaboratory acquired a Velodyne LiDAR Puck Lite and a DJI Matrice 600 Pro. We are currently in the process of developing a workflow for data creation using LiDAR technology (stitching and processing) and have initiated test flights with the Matrice 600 Pro.

Processing Review and 3D Reconstruction
The majority of our processing and 3D reconstruction activity to date has been based on aerial photogrammetry. "Photogrammetry is the art, science, and technology of obtaining reliable information about physical objects and the environment through processes of recording, measuring, and interpreting photographic images and patterns of recorded radiant electromagnetic energy and other phenomena (Wolf & Dewitt, 2000;McGlone, 2004in Aber et al., 2010." Aerial photogrammetry requires a combination of flight planning (at a predetermined altitude and applying the optimal image overlap and camera angle), imagery acquisition and processing workflows for the creation of data products. We use a mobile application called Pix4DCapture to plan flight missions. The software also has the capability to interact directly with the drone and execute autonomous flights to capture data which provides consistency in the imagery acquisition far beyond the capability of manual human controlled flights. Post-flight, the data is processed with the desktop application, Pix4D Mapper. Processing: Using Pix4D software, we are able to create the following data products using the aerial photogrammetry process: Orthomosaics, Digital Elevation Models (DEMs), 3D Point Cloud datasets (.las file format), and 3D Textured Meshes. While processing with Pix4D software, an environmental scan of related processing technologies was conducted which led us to ESRI's Drone2Map software. This software is based on Pix4D technology and produces comparable outputs. In addition, the resulting datasets are ArcGIS ready for users who choose to conduct analysis using the ArcGIS suite of software. Drone2Map is also a cost effective option for those institutions that have existing ESRI site licenses. We are in the process of comparing processing results between Pix4D and Drone2Map to ensure the production of best quality data products.

East and West Chinatown Heritage Project
Professor Linda Zhang is a faculty member in the Ryerson School of Interior design. She began to look into a project that focused on preserving the heritage of Toronto's East and West Chinatowns. After becoming a member of the Collaboratory, she worked with Jimmy Tran to explore various methods to create digital 3D models of architectural structures of the Chinatowns in Toronto. Through this work, the use of drone technology was identified as an extremely effective method for systematic image acquisition enabling aerial photogrammetry utilizing software such as Pix4D Mapper to build 3D models. Professor Zhang was able to further process these data to enable 3D printing in the design of a Build Your Own Chinatown board game. Although photogrammetry was successful in Chinatown East, Chinatown West posed some challenges as there were many large trees blocking buildings. One work around was to collect data during leaf-off periods in early spring but continual development of LiDaR stitching would be the optimal solution. Figures 2 and  3 demonstrate a 3D model of the "Chinese Arch" in Chinatown East.

Ryerson University Campus Core Revitalization Project
A component of Ryerson's Public Realm Plan focusing on the redevelopment of Gould Street and Victoria Lane commenced in 2019. In order to document the phases of development, periodic flights have been conducted to capture these changes to the campus over time. Data products created include Orthomosaics, 3D Textured Meshes and 3D Point Clouds. The intention is to document and preserve this piece of Ryerson's history and to make these datasets openly available to the research community. Figure 4 demonstrates a 3D Textured Mesh created using the photogrammetry process.

3D Reconstruction of the Urban Forest -Glen Stewart Ravine and Allan Gardens, Toronto
In collaboration with Dr. Andrew Millward (Department of Geography and Environmental Studies, Ryerson University), we have been exploring whether aerial photogrammetry is a viable approach for creating 3D representations of the Urban Tree Canopy. To date we have conducted flights in both the Glen Stewart Ravine and Allan Gardens in Toronto. We have experimented flying at various altitudes and applying different camera angles to assess whether aerial photogrammetry is capable of producing accurate results. Based on our observations, we have determined that LiDAR technology will be required to produce the desired 3D Point Cloud datasets.

Course Curriculum Enhancement -GEO 542: Introduction to Remote Sensing
In collaboration with Dr. Wayne Forsythe (Department of Geography and Environmental Studies, Ryerson University), we delivered a session introducing students in GEO 542 -Introduction to Remote Sensing to drone technology. The session included a live demonstration of drone functionality, including both manual and autonomous flights. The demonstration was followed by a lecture in the Collaboratory which focused on the regulations and legal requirements for flying Remotely Piloted Aircraft Systems (RPAS) in Canada and showcased selected data outputs resulting from aerial photogrammetry. Following our session, a participant designed a project involving the use of drones in agriculture and was recently awarded a Creators Grant via the Library's Digital Media Experience Lab.

Visualization Virtual Reality (VR) Environment
The recent progress made in VR technology has provided exciting options to visualize data. The Ryerson Library Collaboratory developed an immersive VR experience where users/players are able to walk around East Chinatown in Toronto as if they were actually there. This was done using the Unity Game Engine on the HTC Vive VR platform. The 3D models from the Chinatown Heritage project created by Pix4D Mapper were created from images captured by drones. This project was initiated as a proof of concept to show that powerful visualization can be created with relative ease using simple tools available to our researchers.

Immersive 360 Degree Projection Environment
Ryerson University Library is in the planning stages of building an Immersion Studio. Based on the Igloo Immersion 360 degree Projection System, the Library's Immersion Studio is going to develop extensible models and programs for the development and dissemination of digital research and scholarship in a large-scale and immersive visualization environment. This technology infrastructure will provide faculty opportunities for pedagogical innovation and enrich students' learning experience. With the Immersion Studio, the Library's capacity to support teaching, learning and research will be expanded to a new level. We are in the process of working with 360 degree cameras and related technologies, and will pilot content for the Immersion Studio based on our drone imagery acquisition and processing outputs to date.

3D Reconstruction using LiDAR Sensor
Aside from photogrammetry, the other method for 3D reconstruction is to use a LiDAR sensor to scan the environment. A LiDAR sensor uses a spinning laser to measure points in 3D space and produces 3D point clouds that are more accurate than those created using photogrammetry. Algorithms such as Iterative Closest Points (Besl & McKay, 1992) serve as a basis for a general technique called Simultaneous Localization and Mapping (SLAM) (Nuchter et al., 2005) to align each individual scan together to produce a large 3D model. In outdoor environments where the sensor is visible to Global Positioning System (GPS) satellites, industrial grade GPS receivers can be used to create a geo-located Point Cloud.
Up until a few years ago, LiDAR technology has been prohibitively expensive, requiring significant financial investment to employ the technology. As a result, activity was typically limited to large corporations or institutions given the high cost (above $100,000). Given the recent progress in robotics and accelerated development of autonomous vehicles (self driving cars), the cost of LiDAR technology has decreased significantly to a few thousand dollars (Puck LITE, 2020). Albeit, more affordable now, LiDAR is still much more expensive than photogrammetry and a complete turn-key solution for LiDAR scanning and 3D model generation still requires an investment in the tens of thousands of dollars. The Ryerson Library Collaboratory is in the process of developing a cost effective way for researchers to have access to this technology by developing software using open source research projects.

Library Drone Lending Vs. Learning Program
As drone technology has become more affordable in recent years, it is becoming a more prominent component of the research process. Before pilot certification became a requirement to legally operate drones in Canada, Ryerson Library investigated the potential for a library drone lending program. In this scenario, the library would loan the hardware to researchers who could demonstrate competence with operation of the RPAS. Since June 1, 2019, licensing requirements have complicated the potential for this initiative. In addition to required pilot certification, drones must be marked and registered to licensed operators, further complicating the loan process. Furthermore, the potential for accidents and/or aircraft malfunction is greater with novice operators, thus introducing financial risk related to drone repair and/or replacement. Finally, any library loaning drones must ensure that drones owned and operated by the university in compliance with Canadian Aviation Regulations are covered under the institutions' General Liability policy in addition to assessing the deductibles associated with property damage policies. These challenges have redirected our vision from Library Drone Lending to the development of a Library Drone Learning program.
The scope for our Library Drone Learning Program includes guidance for patrons interested in acquiring a Drone Pilot Certificate for Basic Operations. When demand exists, consultation services also extend to Advanced Operations once the basic certification has been obtained. Beyond licensing, the Mavic Mini provides an excellent opportunity for flight training given that no RPAS certification is required to operate this model. We are currently supporting curriculum in undergraduate and graduate programs and collaborating with faculty and students across disciplines. As the service grows we will assess sustainability which may introduce a cost recovery model for our time that can support further development or the acquisition of new hardware.

Closing Remarks
Our work to date has uncovered many challenges related to the development of a research service employing drones and 3D modeling technologies. Support for such a service requires a specialized knowledge base and significant time commitment to obtain the necessary certification to operate drones. That said, the Library is very well positioned to provide this expertise given the interdisciplinary nature of its support services. Our local community of researchers interested in the application of drones continues to grow based on diverse research agendas and we hope to expand this community externally as we continue to grow our knowledge base.