Multimodal Sensing and Artificial Intelligence: Technologies and Applications III

Transport assets are progressively more exposed to several issues, including climate change, vulnerability to natural hazards as subsidence, landslides and seismic phenomena, to mention a few. They can affect the structural integrity of infrastructures causing damages and deteriorations. In addition, the progressive ageing of assets and the degradation of materials, sometimes accelerated by the severe environmental conditions, are not contrasted by effective maintenance actions, because of the inappropriate and insufficient allocation of financial resources. Novel efficient and cost effective methods are needed for large-scale survey of infrastructures, and simultaneous detailed identification of local damages. They support the prioritization of the most effective countermeasures. We propose a method based on Data-Fusion approach, merging multi-source and multi-scale information acquired by multisensors systems. It enhances the interpretation of fused data under a holistic framework. For this purpose, processing methods of spaceborne data, such as Multi-temporal SAR Interferometry, are used to detect displacements of transport assets, with millimetre accuracy. This technique can be effectively complemented by more detailed data detected by UAVs and Ground Based Non-Destructive Testing Methods, including laser scanner surveys and by prospecting data collected by Ground Penetrating Radar. All these data are simultaneously fused and digitally integrated into a comprehensive and high resoluted Digital Twin, providing a useful tool to support operators and public authorities to identify and prioritize maintenance and rehabilitation actions. According to this holistic framework, the introduction of the Digital Twin, which replicates a real asset in a virtual or augmented reality environment, represents a relevant and strategic tool or facility to tackle effectively and efficiently the maintenance issues, enhancing resilience and operating life of the infrastructures. This study shows an experimental application of the proposed methodology for mapping wide areas affected by potential criticalities for the identification of the main vulnerabilities of the infrastructure. The time history of the vertical displacements of the infrastructure and their trends are generated through SAR data processing. The Digital Twin is implemented fusing spatial, aerial, ground-based and geophysical surveys and using also augmented reality tools, in order to have the best interface for visualizing the twin in the real environment. This allows the smart evaluation and monitoring of the infrastructure, this means effective, resoluted, efficient, holistic and up-to-date.

Building Information Modeling (BIM) is a software-based parametric design approach that enables complete interoperability between all of the various actors engaged in a design or management process. Notwithstanding It was particularly developed for construction projects, it has since been applied to a variety of fields, including the design of transportation infrastructures and, more recently, cultural heritage. It has primarily been used in this area to improve the accuracy and efficiency of activities aimed at stabilizing and restoring historical structures. The purpose of the current study is to show how BIM is implemented by multisensors date and how the use of BIM can significantly raise the quality of virtual reconstructions and digital valorizations of historical structures, particularly when the rate of conservation is low. In spite of the fact that modern digital reconstruction models are frequently examined from an archaeological standpoint, their structural coherence has never been examined. This implies that many virtual models are likely to depict historically reliable structures that, due to their lack of structural logic, would not support their weight according to their geometric features and construction methods/materials. The research suggests a novel BIM-based methodology capable of both guiding and structurally testing archaeological reconstruction theories. The following procedure can serve as a broad representation of the model: 1- Multisensors survey of the emerging: gathering information from cursory historical investigations (topographic data, laser scanner, aero photogrammetry, satellite images) 2- Multisensors survey of the hidden: gathering information from ground-penetrating radar, electrical tomography, and magnetometry hypogeal scans; 3- Mechanical characterization: collecting data on the materials used in the discovery and confirming their mechanical properties through load stress tests; 4- Virtual reconstruction: the formulation of a potential hypothesis related to structural and morphological characteristics that are known to have existed during the referred historical eras; 5- Structural test: engineering and structural verification of the proposed theory using finite element algorithms and graphic statics techniques. In the framework of the Project BIMHERIT, funded by Regione Lazio, the proposed methodology was tested at a preliminary stage on the archaeological site of the Villa and Circus of Maxentius along the Ancient Appian Way in Rome. All planned activities have been shared with and authorized by the Sovrintendenza Capitolina ai Beni Culturali (DTC Lazio Call, Prot. 305-2020-35609).

Hot boxes, which refer to overheated rail-road car wheels and bearings, pose a significant threat to railway operations. Failure to detect and address hot boxes promptly can lead to catastrophic accidents such as derailments and fires. The main contribution of this paper lies with the development of an automatic hot box detection, tracking and counting method using the IR thermal cameras. The method combines the YOLO algorithm with the Kalman filter (KF) as a tracker. Original dataset was constructed from experiments conducted at different times of the day. The method points the way towards efficient and low-cost hot box detectors.

The timely detection of leakage in water mains is an issue that is relevant to the sustainable and efficient use of natural resources and the prevention of environmental hazards and risks for citizens. Consequently, the development of non-destructive techniques capable of detecting and localizing water leaks in buried pipelines is of huge interest. In this contribution, we present an artificial intelligence tool to perform automatic leakage detection from ground penetrating radar tomographic images. Ground penetrating radar is a prominent technology for subsoil inspection based on the remote interaction of microwave signals with buried anomalies, but its results require expert-users and are prone to subjective interpretation. This can be counteracted by processing raw-data using microwave tomography algorithms, which are capable of delivering more easily interpretable images. However, tomographic images can be still very difficult to interpret when the assumptions underlying the algorithm fail and therefore do not lead to conclusive results. To overcome this issue, we cast the leakage-detection problem as an image segmentation task, in which the popular convolutional neural network U-NET is trained to turn tomographic images obtained from raw-data processing into binary images clearly depicting the location of the leaks. Preliminary results with full-wave synthetic data confirm the potential of the proposed approach.

AAutomated systems increase their requirements in all fields, tightening their performance requirements in aspects like reliability, and ease of manipulation. Within this communication, we will present the development of a compact perception unit which includes a 3D lidar, RGB and thermal imaging for advanced perception purposes. The proposed unit intends to solve the usual hardware problems that software developers intend to solve in field applications. The basic features and performance of the system will be presented, and the applicability of the multimodal sensing approach presented to different applications in security, autonomous vehicles, and other application areas will be overviewed with examples.

Detecting defects that may arise in weldings using non-destructive tests is an area of active research. However, non-destructive tests require a significant effort from domain experts, whose evaluations may also be affected by subjectivity. To address these challenges, this work proposes an end-to-end framework for anomaly detection in linear aluminum welding, feeding its three-dimensional representation to an autoencoder trained to associate a small reconstruction error on non-defective samples, while providing large errors when fed with defective ones. Preliminary results show the effectiveness and feasibility of the proposed method, which can achieve an accuracy of over 80%.

Direct Energy Deposition (DED) is an additive manufacturing technology that allows for both the realization of complex shape objects and the repairing of damaged or worn metallic components. A high-power laser beam spans the substrate surface while metallic powder is added by nozzles, creating overlapping solid tracks and thus building up parts in a layer-wise fashion. Despite fine parameters optimization, the process instability may lead to layer height variations that could accumulate over time affecting the stand-off distance between the printing head and the substrate and impairing the printed part geometry. In the present work, an innovative process monitoring approach has been investigated. Sensors such as a pyrometer and RGB camera have been integrated into a DED machine, to measure the radiation emitted by the melted material at different spectral bandwidths. A neural network approach has been derived to estimate the stand-off distance and highlight possible anomalous situations. The training dataset has been acquired by printing tracks in several conditions. The proposed solution has been validated by comparing the geometry estimated on defective multilayer parts with the measurements provided by a high-resolution 3D scanner. Preliminary results show that the proposed method is a promising approach to prevent critical stand-off deviations.

I will present a photonic sensor that can be used for remote sensing of many biomedical parameters simultaneously and continuously. The technology is based upon illuminating a surface with a laser and then using an imaging camera to perform temporal and spatial tracking of secondary speckle patterns in order to have nano metric accurate estimation of the movement of the back reflecting surface. The capability of sensing those movements in nano-metric precision allows connecting the movement with remote bio-sensing and with medical diagnosis capabilities. The proposed technology was already applied for remote and continuous estimation of vital bio-signs (such as heart beats, respiration, blood pulse pressure and intra ocular pressure), for molecular sensing of chemicals in the blood stream (such as for estimation of alcohol, glucose and lactate concentrations in blood stream, blood coagulation and oximetry) as well as for sensing of hemodynamic characteristics such as blood flow related to brain activity. The sensor can be used for early diagnosis of diseases such as otitis, melanoma and breast cancer and lately it was tested in large scale clinical trials and provided highly efficient medical diagnosis capabilities for cardiopulmonary diseases. The capability of the sensor was also tested and verified in providing remote high-quality characterization of brain activity.

The modern development of microelectronics and the transition to get new technologies have made it form to minimize video data recording devices. Using mobile or small fixation complexes has found its application in many areas. The most popular technology has become a technology that allows you to expand the possibilities of human perception using other electromagnetic ranges. The most popular and used for practical tasks are the ranges of X-ray radiation (dangerous for humans) and infrared radiation (safe for humans at low powers). The near IR spectrum allows you to see in the dark or foggy conditions, the far IR spectrum allows you to form thermal portraits of objects. The imperfection and presence of inhomogeneities in the sensors adds a noise component to the generated data. Most often, to eliminate noise, the averaging procedure over a series of frames is used, but the relative low performance does not allow this procedure to be performed in real time. The problem of primary data processing for thermal imaging cameras is still relevant. To eliminate the noise component in infrared images, the paper proposes a method of inter-frame data alignment, built on the operations of pre-processing and highlighting the contours of objects and based on the following basic stapes: at the initial stage, the system operation parameters are determined, the frame size is read, the color field depth and speed fixing frames; the operation of selecting local areas with minimization of objects allows you to create a mask of objects and localize subsequent methods; determining the construction of inter-frame pixel shift vectors; accumulation\formation of color intensity change data; data filtering based on a multi-criteria method based on the application of the criterion of the standard deviation of the results and the generated estimates, as well as the standard deviations of the final differences of the second order; frame processing of images by a two-dimensional multi-criteria method with preservation of object boundaries; framing the output image. On the set of test data obtained by the seak and flir cameras, examples of processing objects of simple shapes are given. Examples of visual reduction of the noise component on frames of test images are shown. Examples of error reduction in the formation of the proposed approach on synthetic sequences with superimposed Gaussian noise are given.

For both economic and research purposes, accurate information on forest health and composition is of paramount importance. However, using conventional methods of surveying can be very time consuming for such an inventory. We present an innovative approach for forest inventory combining UAV-borne LiDAR, multispectral aerial imagery and terrestrial point cloud data measured with a handheld laser scanner. The result of this procedure are radiometrically enhanced 3D point clouds containing detailed information about the forest structure above and underneath its canopy. This provides a versatile basis for further processing such as tree species recognition, e.g., using convolutional neural networks.

This study investigates the potential of a consumer-grade infrared stereo camera, i.e. the Intel RealSense D435, to automatically extract crop status information, such as Normalized Difference Vegetation Index (NDVI), in arable and permanent crops. The sensing device includes two infrared (IR) sensors for depth calculation and one colour sensor, which provide, for each point of the scene, both IR and visible light information thus making it possible pixel per pixel NDVI estimations. Measurements were performed on various arable crops including corn (Zea mays) and barley (Ordeum vulgare) and on two vine varieties, Freisa and Malvasia, and were compared to measurements taken by a Trimble GreenSeeker handheld crop sensor. Results show that although the RealSense camera tends to underestimate NDVI values compared to the GreenSeeker, there is a good correlation between the two sensing devices with squared correlation coefficient r2 = 0.68 and root-mean-square error RMSE = 0.07. The fitted regression equation was successively applied to correct new camera observations and compared with the GreenSeeker output, showing good agreement with RMSE of 0.07 and mean squared error (MSE) of 0.01. The use of the RGB-D camera to simultaneously provide NDVI and canopy height measurements by a farmer robot is also demonstrated in a Malvasia vineyard field, showing that the proposed system can be effectively adopted for fully automated plant-scale monitoring of high-value crops.

A microcontroller-based multi-sensor system using ultrasonic range sensors and radar sensors for sensor data fusion is presented. Using this system as a warning system can detect and prevent forklift accidents in the industry: The system uses the combined sensor data to detect objects’ distance, angle, and speed and calculate object trajectories to determine the risk of each moving object in the detection area. Because the dimensions of the microcontroller-based warning system are very small, it can be integrated into a warning vest for a factory worker. If the system detects and categorizes objects as risk, the factory worker is noticed by vibrotactile feedback about the location of the risk object. The paper deals with the introduction of the system followed by accuracy study and performance test.

Automation of production processes using robots is a priority for developing many industrial enterprises. Human-machine interaction is a key component of such control infrastructure. The proposed algorithm is a four-stage procedure: (a) fusion information from multimodal sensors based on the quaternion model, (b) image preprocessing using a 3D Gabor filter, (c) a descriptor calculation using 3D local binary dense micro-block difference with skeleton points, and (d) classification. The proposed algorithm is based on capturing 3D sub-volumes located inside a video sequence patch and calculating the difference in intensities between these sub-volumes; for intensified motion, used the convolution with a bank of 3D arbitrarily oriented Gabor filters and calculating 3D local binary dense micro-block difference. A program was developed for transmitting information about the target points of the robot's movement through a virtual TCP / IP port and a script for working out the target points in the simulation environment. To test the effectiveness of the proposed algorithm, we simulate an action recognition system for Human-robot collaboration in the RoboGuide environment.

Multispectral LiDAR is an emerging active remote sensing technique that combines distance and spectroscopy measurements on light reflected from the surface at the respective measurement point. It is known that the reflectance spectrum can be used for material classification. However, the spectrum also depends on other surface parameters, particularly surface roughness. Herein, we propose an extension of multispectral to polarimetric multispectral LiDAR and introduce polarized and unpolarized reflectance spectra as additional features for classifying materials and roughness. We demonstrate the feasibility and the benefit using a bench-top prototype instrument which allows acquiring standard, polarized and unpolarized reflectance spectra, in addition to distance, in 33 spectral channels with 10 nm bandwidth between 580 and 900 nm. We analyze and interpret the raw spectra obtained from measurements on test specimens consisting of five different materials (PE, PVC, PP, sandstone, limestone) with two different levels of surface roughness. Using a linear support vector machine (SVM) we demonstrate the potential of the different features for independent material and roughness classification. The results indicate that the unpolarized reflectance spectrum increases the material classification accuracy by 50% as compared to a standard spectrum, and that the polarized spectrum actually allows classifying roughness. We interpret the results as a strong indication that multispectral polarimetric LiDAR enables deriving practically relevant additional information on surfaces with high spatial resolution through remote sensing.

Carbon Fiber Reinforced Plastic (CFRP) is an important material in manufacturing, in particular in automotive and aerospace industry. Its relevance is due to its lightness, resistance and rigidity. In particular the lightness/resistance ratio is the property that will give to CFRP a favourable future in more and more applications, especially where the lightness and resistance are essential. On the other end CFRP is expensive and difficult to produce. During the production process of CFRP’s artefacts some defects may occur (e.g. material inclusions or bubbles), making the component not usable. When applicable, CFRP repair process is structured in: defect removal (scarfing), cleaning and patch application. Actually, scarfing is a slow process manually performed by human operators, strongly dependent on the workers’ skills, that generates a lot of toxic nano dusts. The present paper deals with the implementation of a (semi)automatic robotic cell to repair large CFRP aeronautic components. The robotic cell is composed by a collaborative robot equipped by a custom scarfing tool due to the context-specific constraints to be implemented. Under these constraints, the machining got is not as perfect as in a traditional milling process, thus requiring additional quality control and inspection after scarfing takes place. The paper describes the designed scarfing tool, examines the trajectories generated to execute the scarfing and illustrates some preliminary results in terms of accuracy of the shape got by the designed tool.

This paper presents a preliminary study for evaluating the quality of welds in thermomagnetic switches using 3D sensing and machine learning techniques. A 3D sensor based on laser triangulation is used to gather the point cloud of the component. The point cloud is then processed to extract hand-crafted signatures for binary classification: defective or non-defective component. Features such as Gaussian and mean curvatures, density, and quadric surface properties, are used for building these significant signatures. Different machine learning models, including decision trees, Support Vector Machines, k-nearest neighbors, random forests, ensemble classifiers, and Artificial Neural Networks, are trained using the built signatures to classify the weld as defective or non-defective. Preliminary results on actual data achieve high classification accuracy (>80%) on all the tested models.

Monitoring and maintaining existing bridge stocks is a critical task. To reduce the required effort, automatic techniques for object detection were proposed. However, many of the existing algorithms either deliver poor accuracy (i.e., template-matching algorithms) or are computationally expensive (i.e., two-stages detectors). To overcome these issues, this work proposes a single-stage damage detector based on YOLOv5. To demonstrate the effectiveness of the proposed approach, a database composed by 6835 images from several bridges with seven classes of defects has been gathered. The overall results show a mean average precision above 60%, with high precision and recall.

Raman hyperspectral imaging allows the rapid screening of surfaces potentially contaminated by toxic chemicals. We present an application of multivariate models to the detection of target chemical signatures in Fourier-Transform Raman hyperspectral imaging. Working directly on the interferograms (Fourier domain) our method was developed to address two main challenges. First, the presence of possibly strong substrate signal overlapping with the target signal. Second, the variation in measured interferograms across the field of view requiring the use of locally trained models. Detection results obtained on a set of target compounds deposited on polymer substrates are presented.

Mapping of small and shallow water bodies is still primarily done manually. The Fraunhofer IPM developed and evaluated a compact airborne laser-scanning system which weighs under 4 kg and targets topographic and bathymetric surveys. Our sensor simultaneously emits green and infrared light. Combined with full waveform signal analysis, this allows for precise separation of water surface, seabed, and shoreline. A downward-facing palmer scanner deflects the beam in a near-elliptical shape. The system has been tested and evaluated at a quarry lake in turbid water conditions. The result is a high-resolution ground model demonstrating the benefits of a compact bathymetric scanner.

Caterpillars have been causing havoc to agriculture due to feeding aggressively on the foliage of the crops. The current methods of pest control like sticky traps or pheromone traps work on adult moths, but not on foraging caterpillars. Chemical means like pesticides are effective, but chemical residues on crops are concerning. Therefore, this study aims to primarily track and estimate the 3D position of the caterpillars in orchards in real-time. To this end, we have employed the state-of-the-art object detector YOLOv7, combined with Simple Online and Realtime Tracking with a Deep Association Metric (DeepSORT) algorithm. This combined approach, when compared to merely YOLOv7, has improved detection up to 25%, courtesy of the DeepSORT embedded tracker. The RGB-D camera is utilized for this work is Intel Realsense D405. For the training data, 2,000 images captured in a jujube orchard with varying exposure, occlusion, and wind conditions were used. Inference was done from completely new images in real time. In the experiments, the YOLOv7+DeepSORT approach makes detections within 14 ms per frame, with an average detection rate of 85%, indicative of is real-time applicability in orchards. The smallest object (around 2-cm length caterpillar) is recognized around 21×12 pixels, which is at a distance of 35 cm from the camera. Thus, this development of YOLOv7+DeepSORT approach can be integrated with technologies like laser robot arms, that can pick the caterpillars, or even into stand-off techniques like laser pest targeting, which can help eradicate the pest problems in a physical manner efficiently.

This study addresses a 2D scalar electromagnetic inverse source problem by using a deep neural network-based artificial intelligence technique. Specifically, the Learned Singular Value Decomposition (L-SVD) approach based on hybrid autoencoding is adopted. The main goal is to reproduce the singular value decomposition through neural networks and compare the reconstruction performance of L-SVD and truncated SVD (TSVD) in the case of noiseless data, which represents a reference benchmark. The results demonstrate that L-SVD outperforms TSVD in terms of spatial resolution.

Multimodal sensing is presented based on the measurement of density oscillations in swirl-stabilized flames, where density is coupled to physical quantities like heat release rate or sound pressure and advection velocity. The measurement approach is a high-speed camera based interferometer combined with a multi-camera background oriented Schlieren system, both enabling line of sight detection and therefore require solution of the inverse problem by tomographic reconstruction. Those reconstructions are realized using a neural network, resulting in the three-dimensional distribution of local thermoacoustic oscillations. Finally, the advection velocity of those oscillating vortex structures is calculated by image correlation.

We demonstrate a supercontinuum-based hyperspectral laser scanning technique that provides high-precision distance measurements of natural surfaces along with their reflectance signature over the broad spectral range of the supercontinuum (SC) output. The SC used in our experiments is spectrally broadened to 570-970 nm from a 780 nm mode-locked femtosecond laser. Distance measurements are carried out by monitoring the differential phase delay of the intermode beat notes obtained from direct photodetection of the SC, while the backscattered reflection spectrum is acquired using a commercial spectrometer. We achieve a single-point range precision below 10 μm on natural targets (gypsum board and leaves of a plant used herein) placed at a stand-off distance of 5 m. Our results demonstrate the acquisition of hyperspectral point clouds together with sub-mm range noise on the scanned surface. This range performance is comparable to commercial state-of-the-art terrestrial laser scanners which traditionally employ a monochromatic laser source. We show the benefit of enhanced range precision toward correctly estimating the surface orientation and for radiometric calibration of the acquired intensities. Initial results illustrate the direct 3D mapping of spectral data of plant leaves with a reduced angle of incidence-related bias, highlighting new opportunities for future research into remote sensing of vegetation.

The interest in LiDAR imaging systems has recently increased in outdoor ground-based applications related to computer vision. However, for the complete settling of the technology, there are still obstacles pending to be solved, being its use in adverse weather conditions one of the most challenging. Under the presence of bad weather, such as fog, the system performance is heavily altered and the quality of the detection becomes severely degraded. We propose to take advantage of the temporal digitization capabilities of a LiDAR setup combined with the use of polarimetric information, to try to improve the detection. We designed and constructed a LiDAR imaging system with outstanding characteristics related to optoelectronic modifications, in particular including the digitization of light pulses and the addition of circularly polarizing optical elements for polarimetric discrimination. To analyse the performance of the system, it was tested in a macro-scale fog chamber, and the acquired signal was analysed and characterized using cross-polarized detection (cross-channel) and co-polarized detection (co-channel). From the data obtained, we could identify and study different relevant phenomena which were only exposed through temporal digitization and which might enable optimized signal processing features. The backscattering signal of light that first interacts with the media is usually responsible for the saturation of the sensor but it may be easily filtered out using cross-channel. False-positive points that appear due to the scattering properties of the media are also reduced in cross-channel. We also demonstrated that cross-channel data shows increased range and post-processing of the point cloud, including detection and segmentation of objects embedded in the fog, is easier. So, we conclude that a system using a cross-configuration polarized light for detection could help to decrease the influence of backscattering from turbid media without compromising the signal returning from the objects of interest in most cases.

Non-contact methods can expand the application scenarios of blood oxygen measurement with better hygiene and comfort, but the traditional non-contact methods are usually less accurate. In this study a novel non-contact approach for measuring peripheral oxygen saturation (SpO2) using deep learning and near-infrared multispectral videos is proposed. After a series of data processing including shading correction, global detrending and spectral channel normalization to reduce the influences from illumination non-uniformity, ambient light, and skin tone, the preprocessed video data are split into half-second clips (30 frames) as input of the 3D convolutional residual network. In the experiment, multispectral videos in 25 channels of hand palms from 7 participants were captured. The experimental results show that the proposed approach can accurately estimate SpO2 from near-infrared multispectral videos, which demonstrates the agreement with commercial pulse oximeter. The study also evaluated the performance of the approach with different combinations of near-infrared channels.

This talk presents a deep learning approach to phase unwrapping, which is a troublesome step in the sinogram creation in holographic tomography reconstruction. Through implementation of Attention U-Nets and Residual Blocks, phase is successfully denoised and unwrapped. Experiments show that the proposed solution is capable of unwrapping irregular, noisy and complex phase images with equal or, at times, superior performance. The solution has also been implemented into the holographic tomography reconstruction, which enables evaluation on the refractive index distribution of real specimen, against conventional State-of-the-Art solutions.

In the recent years, Holographic Imaging Flow Cytometry has been demonstrated as a promising label-free and quantitative technique for the high-throughput analysis at the single-cell level. By recording hundreds of digital holograms per cell while it is flowing and rotating along a microfluidic channel, the 3D tomogram of its refractive indices can be reconstructed from the corresponding phase-contrast maps. Here we show that a fully convolutional end-to-end neural network can speed up the phase retrieval process, thus reducing the computational time for the tomographic reconstruction from tens of minutes to few seconds, while providing high fidelity and small memory footprint.

TIn recent years, composite materials have found numerous applications due to their mechanical characteristics and properties. The present work is focused on the characterization of composite materials using nondestructive techniques (NDT). Shearography and t thermography are used as nondestructive methods. The former, is an optical interferometric method for the detection of surface or sub-surface defects, the latter is a diagnostic technique for determining surface temperature and to understand the health status of the investigated object. Their use offers advantages related to visualization and testing of end products, noncontact nature, nondestructive and areal operating principle, rapid response, high sensitivity, resolution and accuracy

Lensless microscopy is an emerging imaging modality that overcomes the inherent limitation of conventional lens-based optics, especially in terms of imaging throughput, functionality, and cost-effectiveness. Pixel super-resolution phase retrieval serves as the key underlying technique for reconstructing high-resolution holographic images from the raw measurements. In this talk, we revisit lensless microscopy from a computational imaging perspective. A unified mathematical framework is established and the encoding and decoding mechanisms of the phase and subpixel information are analyzed. Regularization and Nesterov’s momentum techniques are introduced to speed up the data acquisition and reconstruction procedures, respectively. The proposed algorithms are verified through a proof-of-concept lensless on-chip microscope. We experimentally demonstrate the capability of pixel super-resolution phase retrieval techniques in revealing the subpixel and quantitative phase information of complex biological samples.

Microplastics (MPs) are abundant and not degradable pollutants of marine waters. Here we show how Digital Holography (DH) coupled to machine learning can be used to identify and characterize MPs in water samples. We show that features extracted from the wrapped phase-contrast map are key elements for effective identification of microplastic items of various shapes and sizes. We apply the fractal geometry framework to phase-contrast images to discern between several classes of microplastics and microalgae. We also show that a polarization-resolved holographic flow cytometer can use plastic birefringence properties to identify different synthetic and natural microfibers with material specificity.

I will present recent results obtained by table-top EUV ptychography microscopy based on a compact high-harmonic generation source providing a high coherent power at 13.5 nm wavelength. Sub-20 nm resolution is demonstrated by utilizing a combination of structured illumination and an advanced iterative phase retrieval engine. We achieve nanoscale and material-specific EUV imaging and apply this technique to a variety of samples from life and material sciences. EUV ptychography combines high resolution, strong material contrast, reasonable penetration depth and easy sample preparation. It thus has the potential to bridges the gap between visible-light and electron microscopy in many application fields.

Achieving clear vision through smoke and flames is a highly pursued goal to better manage intervention priorities and to allow first responders operating safely during fire accidents. Here we show different classes of active far-infrared systems to image static/moving targets through fire with different imaging performance and field-portability characteristics. Low-coherence infrared systems and high-coherence holographic sensors will be discussed. We show that a pre-trained convolutional neural network can detect the presence of a person hidden behind fire in real-time, accurately, even when the system is not able to reject the flame contributions in full, being suitable for video-surveillance applications.

The assessment of microplastics (MPs) pollution and water quality monitoring raise a lot of attention in recent years. Discriminative methods are highly needed for quick and accurate in situ MP detections. Digital holography records the wavefront information of the objects and contains the morphology, refractive index, and roughness information. Polarization imaging inspects the optical anisotropy of MPs, which is related to their birefringence and material characteristics. In this work, we explore the capability of holographic and polarization imaging for the identification of MPs. The computed features, such as the angle of polarization (AoP) and degree of linear polarization (DoLP), show distinguishable characteristics of MPs. We inspect the method feasibility on MP classification as well as biological and natural particles. The proposed method shows potential use in real-time, non-contact in situ MPs detection and water pollution monitoring.

Weld quality inspection allows the detection of defects that may compromise the quality and strength of the weld. Although visual optical inspection offers lower reliability than other non-destructive methods, it enables weld analysis at a significantly lower cost. In this context, developing machine learning-based algorithms for automatic optical weld quality recognition requires acquiring large amounts of data for training. This entails high costs in terms of time, material and energy required for test preparation. However, one possible approach to tackling the problem with limited datasets is to use synthetic data. Using such data increases the amount and variety of data available to the detection algorithm. With a focus on the context of welding, this paper presents an approach that uses synthetic data as a form of data augmentation to improve the performance of the optical detection of weld seams. Specifically, we propose a generative neural network for semantic image synthesis using a limited starting dataset. The network generates new data instances by receiving as input a semantic map of the image to be represented. Weld defects such as porosity or weld spatter are added to the semantic map so that the network synthesizes corresponding defect images. Analysing the performance on a segmentation network, experimental results show how adding synthetic data to the original data can ensure improvements in network performance.

AI methodologies are strongly impacting on many fields, including computer vision and imaging. However, various limitations have been considered in practical contexts, e.g., paucity of data in space applications, high energy consumption for training, poor generalization in real scenarios, etc. In this work, we will focus on recently proposed techniques of meta-learning to face some of these drawbacks. The concept of multimodal imaging will be analyzed for improving performance and explanations of ML/DL models.

In electronics manufacturing, the inspection of defects of electrical components on printed circuit boards (SMD-PCB) is an import part of the production chain. This process is normally implemented by automatic optical inspection (AOI) systems based on classical computer vision and multimodal imaging. Despite the highly developed image processing, misclassifications can occur due to the different, variable appearance of objects and defects and constantly emerging defect types, which can only be avoided by constant manual supervision and adaption. Therefore, a lot of manpower is needed to do this or to perform a subjective follow-up. In this paper, we present a new method using the principle of multimodal deep learning-based one-class novelty-detection to support AOIs and operators to detect defects more accurate or to determine whether something needs to be changed. By combining with a given AOI classification a powerful adaptive AOI system can be realized. To evaluate the performance of the multimodal novelty-detector, we conducted experiments with SMD-PCB-components imaged in texture and geometric modalities. Based on the idea of one-class-detection only normal data is needed to form training sets. Annotated defect data which is normally only insufficiently available, is only used in the tests. We report about some experiments in accordance with the consistence of data categories to investigate the applicability of this approach in different scenarios. Hereby we compared different state-of-the-art one-class novelty detection techniques using image data of different modalities. Besides the influence of different data fusion methods are discussed to find a good way to use this data and to show the benefits using multimodal data. Our experiments show an outstanding performance of defect detection using multimodal data based on our approach. Our best value of the widely known AUROC reaches more than 0.99 with real test data.

In-field sensing systems for automatic yield monitoring are gaining increasing importance as they promise to give a considerable boost in production. The development of artificial intelligence and sensing technologies to assist the human workforce also meets sustainability needs, which impact the ecological goals of current and future agricultural processes. In this context, image acquisition and processing systems are widely adopted to extract useful information for farmers. Although RGB-D cameras have been used in many applications for ground-based proximal sensing, relatively few works can be found that include depth information in image analysis. In this work, both semantic and depth information from RGB-D vineyard images is used in processing pipeline composed of a decision tree algorithm and a deep learning model. The goal is to reach coherent semantic segmentation of a set of natural images acquired at both long and short distances, using a low-cost RGB-D camera in an experimental vineyard. Depth information of each image is fed into a decision tree to predict the distance of the acquired vines from the camera. Before feeding the deep learning models, the images to be segmented are manipulated according to the predicted distance. The results of semantic segmentation with and without using the decision tree are compared, showing how depth information appears to be highly relevant in enhancing the accuracy and precision of the predicted semantic maps.

We implement the multimodal characterization for bolted joint surface. A reflection digital holographic microscope (RDHM) has been combined with laser triangulation measurement for achieving quantitative micro-deformation description of the connecting region between bolts and metal plate. The calibrated laser triangulation instrument is installed on the electronically controlled translation platform for two-dimensional scanning. A structure extraction algorithm is proposed to realize fast 3D reconstruction of the scanned area. The profiles obtained by triangulation are used, as ground truth, to calibrate the measured reflective holographic morphology. The triangulation will complement digital holography well.

Conventional (CP) and Fourier (FP) ptychography have emerged as versatile quantitative phase imaging techniques. While the main application cases for each technique are different, namely lens-less short wavelength imaging for CP and lens-based visible light imaging for FP, both methods share a common algorithmic ground. CP and FP have in part independently evolved to include experimentally robust forward models and inversion techniques. This separation has resulted in a plethora of algorithmic extensions, some of which have not crossed the boundary from one modality to the other. Here, we describe the principle of reciprocity in ptychography. This principle allows for flexible conversion between CP and FP. Moreover, we present an open source, crossplatform software, called PtyLab, enabling a unified CP and FP data analysis. With this framework, we aim to facilitate and accelerate cross-pollination between the two techniques. Moreover, the availability in Matlab, Python, and Julia will set a low barrier to enter each field.

Many robotics and computer vision scenarios employ fiducial markers as a pose estimation tool to perform closed-loop control or data collection due to their ease of installation and acquisition. Human-robot collaboration and mobile robotics are a few of many examples in which fiducial markers serve as a position estimation tool and where the estimation frequency characterizes the control algorithm employed in terms of performance. The currently available tools are not keeping up with the advancement in acquisition systems, where the raising of image resolution and increasing acquisition rate harden the fiducial marker detection computation. In this work, we propose a method to model and forecast the marker position, leveraging the kinematic properties of the system on which is fixed and estimating its state during the acquisition. The state of the marker, composed of its position and velocity, is updated by an Unscented Kalman Filter(UKF) by using the information extracted from the previous marker estimations and taking into account the kinematic instantaneous configuration. The UKF, knowing the kinematic specification in terms of velocity and positional constraints, will aid the forecast of the marker trajectory. In such a manner, the forecasted position of the marker is used to define a dynamic region of interest that aids the marker localization procedure and improves computation time. The method is then compared to the classical implementation for different fiducial marker systems.