Reservoir Computing with a Snapshot Hyperspectral Camera #

Advanced hyperspectral imaging tools are increasingly used in phenotyping and have been applied to detect changes in plants in response to a specific treatment or phenological state (Citation: Dale, Thewis& al., 2013 Dale, L., Thewis, A., Boudry, C., Rotar, I., Dardenne, P., Baeten, V. & Pierna, J. (2013). Hyperspectral Imaging Applications in Agriculture and Agro-Food Product Quality and Safety Control: A Review. Applied Spectroscopy Reviews, 48(2). 142–159. https://doi.org/10.1080/05704928.2012.705800 ; Citation: Mishra, Asaari& al., 2017 Mishra, P., Asaari, M., Herrero-Langreo, A., Lohumi, S., Diezma, B. & Scheunders, P. (2017). Close range hyperspectral imaging of plants: A review. Biosystems Engineering, 164. 49–67. https://doi.org/10.1016/j.biosystemseng.2017.09.009 ; Citation: Adão, Hruška& al., 2017 Adão, T., Hruška, J., Pádua, L., Bessa, J., Peres, E., Morais, R. & Sousa, J. (2017). Hyperspectral Imaging: A Review on UAV-Based Sensors, Data Processing and Applications for Agriculture and Forestry. Remote Sensing, 9(11). 1110. https://doi.org/10.3390/rs9111110 ; Citation: Khan, Khan& al., 2018) Khan, M., Khan, H., Yousaf, A., Khurshid, K. & Abbas, A. (2018). Modern Trends in Hyperspectral Image Analysis: A Review. IEEE Access, 6. 14118–14129. https://doi.org/10.1109/ACCESS.2018.2812999 . As a result, this is an interesting sensing technology to uncover the computational properties of plants. Two consecutive experiments in growth chambers were set up, in which strawberry plants and four different background materials, serving as controls, were monitored by a snapshot hyperspectral camera in variable conditions of light, temperature and relative humidity. Results indicate that current hyperspectral technologies are not yet suitable for reservoir computing processes. We suspect that limited variation due to the treatment and low spectral resolution and range are the main causes of the inability of the models to extract meaningful information. Furthermore, the models that were only trained on background data also showed good predictive performance.

Part of the data, analysis and results are published in Citation: Pieters, De Swaef& al. (2020) Pieters, O., De Swaef, T., Lootens, P., Stock, M., Roldán-Ruiz, I. & wyffels, F. (2020). Limitations of snapshot hyperspectral cameras to monitor plant response dynamics in stress-free conditions. Computers and Electronics in Agriculture, 179. 105825. https://doi.org/10.1016/j.compag.2020.105825 . This publication discusses a different view of the data and results than what is presented here. The focus is on capturing fast-changing dynamics in plants by means of hyperspectral data.

The Rise of Hyperspectral Imaging in Phenotyping Research #

Hyperspectral imaging sensors capture reflectance in many wavelengths and are increasingly applied in phenotyping research. Figure 5.1 elucidates on the properties of hyperspectral cameras. They are generalised versions of red green blue (RGB) cameras, which capture light in three bands. Hyperspectral cameras can capture hundreds of bands in the visual and near-infrared spectrum (Citation: Adão, Hruška& al., 2017) Adão, T., Hruška, J., Pádua, L., Bessa, J., Peres, E., Morais, R. & Sousa, J. (2017). Hyperspectral Imaging: A Review on UAV-Based Sensors, Data Processing and Applications for Agriculture and Forestry. Remote Sensing, 9(11). 1110. https://doi.org/10.3390/rs9111110 .

This imaging technique has already been applied to various settings that benefit from higher spectral resolutions to detect biotic and abiotic influences on plants (Citation: Khan, Khan& al., 2018) Khan, M., Khan, H., Yousaf, A., Khurshid, K. & Abbas, A. (2018). Modern Trends in Hyperspectral Image Analysis: A Review. IEEE Access, 6. 14118–14129. https://doi.org/10.1109/ACCESS.2018.2812999 . Examples of studies on biotic factors include blight caused by Alternaria solani in potato (Citation: Van De Vijver, Mertens& al., 2020) Van De Vijver, R., Mertens, K., Heungens, K., Somers, B., Nuyttens, D., Borra-Serrano, I., Lootens, P., Roldán-Ruiz, I., Vangeyte, J. & Saeys, W. (2020). In-field detection of Alternaria solani in potato crops using hyperspectral imaging. Computers and Electronics in Agriculture, 168. 105106. https://doi.org/10.1016/j.compag.2019.105106 , late blight caused by Phytophthora infestans in potato (Citation: Franceschini, Bartholomeus& al., 2019) Franceschini, M., Bartholomeus, H., Apeldoorn, D., Suomalainen, J. & Kooistra, L. (2019). Feasibility of Unmanned Aerial Vehicle Optical Imagery for Early Detection and Severity Assessment of Late Blight in Potato. Remote Sensing, 11(3). 224. https://doi.org/10.3390/rs11030224 , or tracking the development of three foliar diseases in barley (Citation: Wahabzada, Mahlein& al., 2016) Wahabzada, M., Mahlein, A., Bauckhage, C., Steiner, U., Oerke, E. & Kersting, K. (2016). Plant Phenotyping using Probabilistic Topic Models: Uncovering the Hyperspectral Language of Plants. Scientific Reports, 6(1). 1–11. https://doi.org/10.1038/srep22482 . Citation: Mahlein (2015) Mahlein, A. (2015). Plant Disease Detection by Imaging Sensors – Parallels and Specific Demands for Precision Agriculture and Plant Phenotyping. Plant Disease, 100(2). 241–251. https://doi.org/10.1094/PDIS-03-15-0340-FE and Citation: Lowe, Harrison& al. (2017) Lowe, A., Harrison, N. & French, A. (2017). Hyperspectral image analysis techniques for the detection and classification of the early onset of plant disease and stress. Plant Methods, 13(1). 80. https://doi.org/10.1186/s13007-017-0233-z provide comprehensive overviews of plant disease detection using imaging sensors and hyperspectral sensors specifically. Studies in which hyperspectral imaging was used to investigate plant responses in interaction with abiotic factors include, for example, detection of green citrus fruits on trees (Citation: Okamoto & Lee, 2009) Okamoto, H. & Lee, W. (2009). Green citrus detection using hyperspectral imaging. Computers and Electronics in Agriculture, 66(2). 201–208. https://doi.org/10.1016/j.compag.2009.02.004 , nitrogen deficiency in sorghum (Citation: Zhao, Reddy& al., 2005) Zhao, D., Reddy, K., Kakani, V. & Reddy, V. (2005). Nitrogen deficiency effects on plant growth, leaf photosynthesis, and hyperspectral reflectance properties of sorghum. European Journal of Agronomy, 22(4). 391–403. https://doi.org/10.1016/j.eja.2004.06.005 , seasonal structural changes and a heterogeneous architecture in an olive orchard (Citation: Zarco-Tejada, Morales& al., 2013) Zarco-Tejada, P., Morales, A., Testi, L. & Villalobos, F. (2013). Spatio-temporal patterns of chlorophyll fluorescence and physiological and structural indices acquired from hyperspectral imagery as compared with carbon fluxes measured with eddy covariance. Remote Sensing of Environment, 133. 102–115. https://doi.org/10.1016/j.rse.2013.02.003 , nitrogen and water distribution quantification in wheat (Citation: Bruning, Liu& al., 2019) Bruning, B., Liu, H., Brien, C., Berger, B., Lewis, M. & Garnett, T. (2019). The Development of Hyperspectral Distribution Maps to Predict the Content and Distribution of Nitrogen and Water in Wheat (Triticum aestivum). Frontiers in Plant Science, 10. https://doi.org/10.3389/fpls.2019.01380 , and drought stress in barley and saxaul (Citation: Behmann, Steinrücken& al., 2014 Behmann, J., Steinrücken, J. & Plümer, L. (2014). Detection of early plant stress responses in hyperspectral images. ISPRS Journal of Photogrammetry and Remote Sensing, 93. 98–111. https://doi.org/10.1016/j.isprsjprs.2014.03.016 ; Citation: Jin & Wang, 2016) Jin, J. & Wang, Q. (2016). Hyperspectral indices based on first derivative spectra closely trace canopy transpiration in a desert plant. Ecological Informatics, 35. 1–8. https://doi.org/10.1016/j.ecoinf.2016.06.004 .

The successes reported in literature, in addition to the non-invasive and high degree of information availability, makes this sensor an excellent first candidate to investigate the applicability of physical reservoir computing (PRC) on plants. Moreover, the resulting dataset can be very interesting since – to the best of our knowledge – there is no research yet on hyperspectral data with respect to photosynthetic activity at high spatial and temporal resolution in the seconds to minutes range in normal conditions.

Experimental Setup #

The experimental setup follows the general setup described in Experimental Design. Additional information on the exact experimental conditions is detailed below.

Measurement Setup #

The experimental setup consisted of a single strawberry plant (Fragaria×ananassa) placed inside a growth chamber. Light intensity, temperature and relative humidity were controlled by a microcontroller board (Dwenguino, Dwengo vzw, Brussels, Belgium), placed outside the growth chamber. The temperature and relative humidity of the growth chamber were controlled using analogue signals and varied randomly between 11°C and 33°C, and 31% and 75% respectively.

A grid of lamps was mounted on the custom frame, consisting of 32 light emitting diode (LED) lamps (MAS LED spot VLE D 4.9-50W GU10 927 60D, Koninklijke Philips N.V., Amsterdam, The Netherlands) and twelve halogen lights (DECOSTAR 51 PRO 50 W 12 V 36° GU5.3, OSRAM GmbH, Munich, Germany). The halogen lights were used as a broadband light source, providing illumination in the visible and infrared range, while the LED lights increased the total photosynthetically active radiation (PAR) while keeping thermal radiation within limits. Graphical depictions of the emission spectra are provided in Supplementary Information, figure 5.13. The light intensity of the halogen lamps was controlled using a digital addressable lighting interface (DALI) controller and bus, while the LED lights were arranged in four sets that could be individually turned on and off. A detailed overview of the grid is depicted in figure 5.2.

A single strawberry leaf was inserted into a transparent leaf chamber of the LI-6400XT photosynthesis system (LI-COR, Lincoln, NE, USA) to acquire gas exchange measurements (transpiration and photosynthesis). The control board also controlled the sampling time steps of the LI-6400XT, using a custom circuit that was connected to the manual sample button on the measurement node. To increase the carbon dioxide concentration in the growth chamber, we used a constant influx of stabilised air. This influx had a carbon dioxide concentration of 500ppm at a rate of 1m³/h. For environment sensing at canopy height, we measured the temperature, light intensity and relative humidity. An external probe (Vaisala 50Y, Vaisala, Helsinki, Finland) was used to measure temperature and humidity. The gas exchange device has a PAR probe to measure light intensity. This device was programmed to recreate the temperature measured using the probe inside the chamber, thus preventing the chamber from heat-up due to the infrared radiation.

In the centre of the lamp grid, a hyperspectral camera consisting of two camera heads (EP-12, {3D-One}, Sulz, Austria) was placed to monitor the plant. One head is sensitive to light in the near-infrared (NIR), while the other is sensitive to the visible range (VIS). We refer to these as H1 and H2 , respectively. Both heads capture 12bit colour information. The sensors were constructed by IMEC (Leuven, Belgium). H1 captures light in 25 spectral bands and has a spatial resolution of 403 × 216. H2 captures light in 16 spectral bands and has a spatial resolution of 504 × 270. A trade-off was made here between spectral accuracy and sampling rate. We set the sampling period to 3s, which was too fast for using a high-resolution line-scanning sensor. The snapshot camera used here can capture images up to 120Hz. One spectral filter was used for each sensor to limit the sensitivity range of the VIS or NIR spectra. The NIR filter was a long pass filter, starting at 675nm and cuts-off at 1650nm (TECHSPEC 675nm 25mm Dia, High Performance Longpass Filter, Edmund Optics, Barrington, NJ, USA), while the VIS filter (SCHOTT BG38, Edmund Optics, Barrington, NJ, USA) starts at 350nm and cuts-off at 645nm. This camera does not support an external trigger source, thus the internal trigger source was configured to sample every 3s. When the response of the cameras were taken into account, this setup observed wavelengths in the ranges 400 to 645nm, and 675 to 1000nm for H1 and H2 respectively. An illustration of the entire setup is shown in figure 5.3.

The spectral peak wavelengths for H1 and H2 are depicted in Supplementary Information: table 5.2 and table 5.3. The spectral resolution of H1 varies between 6 and 25nm, while the spectral resolution of H2 varies between 2 and 14nm.

A camera always makes an indirect observation, so there might be unwanted interference with the measurements. To investigate this, we also included different background materials in the analysis. Four materials were investigated: plywood (hardwood, Van Den Nest, Aalst, Belgium), non-reflective black cotton cloth (Veritas, Kontich, Belgium), grey polyvinyl chloride (PVC) (Scafoam, Scala, Wetteren, Belgium), and Ytong (Xella, Duisburg, Germany). It was not possible to have similar lighting conditions on all four materials simultaneously. Consequently, the experiment was conducted twice. The first experiment used PVC and plywood, while the second used Ytong and cloth. The second experiment was conducted four days after the first one on a different plant. Both plants were grown in the same greenhouse in close proximity and thus experienced similar conditions before the experiment. Each experiment lasted for 100h. Temperature, relative humidity and light radiation were randomly varied, but the same time series was used in both experiments. To generate these random sequences, we drew samples from a uniform distribution.

Data Preparation and Processing #

The setup consisted of two independent data sources: the gas exchange system and the hyperspectral camera. They were not synchronised due to the lack of a common trigger source. Hence, the start of measurement was not aligned. This difference was manually corrected by turning on all LED lights at the start of the experiment. Furthermore, the camera did not always sample consistently. As a result, the time points of the images did not always align with those of the gas exchange system. Both the gas exchange and hyperspectral camera systems provide a sample timestamp. Synchronisation of both sources was achieved using the time points of both sources and linear interpolation to construct sample points on a single time axis.

Time drift between both was negligible for the time span of the experiment because the sample rate was 3s and the duration of the experiment was only 100h. The camera had an internet uplink and was synchronised to a network time protocol (NTP) server. Typical synchronisation offset with NTP-servers are lower than 10ms (Citation: Marouani & Dagenais, 2008) Marouani, H. & Dagenais, M. (2008). Internal Clock Drift Estimation in Computer Clusters. Journal of Computer Systems, Networks, and Communications, 2008. e583162. https://doi.org/10.1155/2008/583162 . The LI6400XT has an accurate internal clock that has at most 0.5s drift after 92 days (Citation: , 2012) (2012). Using the LI‐6400 Portable Photosynthesis System. . Consequently, it is safe to ignore the drift between the two systems.

Photosynthesis is driven to a large extent by light. To maximise the variation in photosynthetic activity of the plant, we varied the illumination intensity. Because a camera requires light to operate, the light was never fully turned off. The lowest PAR value that was applied was 70µmol/m²/s, while the highest was 409µmol/m²/s. The camera was set to automatic exposure to optimise the dynamic range of each image and reduce over- and underexposed areas. In low illumination conditions, there was a considerable amount of noise. Consequently, after aligning and resampling the data, the data was further processed using a 5 × 5 uniform blur filter to reduce noise in the image. Additionally, the data was rescaled and converted to floating-point to compensate for the variable exposure duration.

The leaves of the strawberry plant displayed limited motion from reorientation, as is evident from figure 5.4. Hence, no motion compensation was applied to the image sequence to reduce computational complexity. This also simplified the masking that had to be applied to separate the plant from the background. The mask was manually determined from the dataset. A different set of masks was used for both experiments because the locations of the two plants and four materials differ.

Image data are highly correlated, especially for adjacent pixels. Therefore, we constructed two types of datasets. The first dataset averaged over all spatial locations of each mask to obtain a new data entry. This technique drastically reduced the number of features at each sample point from over 10000¹ to 41 (25+16). To incorporate spatial effects, we constructed a second dataset using subsampling. The images were spatially sampled in a random fashion without duplicates. When a certain location was considered, all spectral bands were included in the dataset. Subsampling sizes ( H1 + H2 ) varied between 3 (1+2) and 200 (78+122) for both experiments. The split was designed to have an (approximately) equal number of spectral features from H1 and H2 . A different set of spatial sample points was selected for each of the plants and each of the materials due the unique mask associated with each substrate.

Subsampling and averaging also helped to deal with the large amount of data that was captured during the trial. Each image from either H1 or H2 had a size of 4.3MB, and there were approximately 130000 valid data points per trial, resulting in a total dataset size for both trials of slightly more than 2TB.

In remote sensing and phenotyping, vegetation index (VI) are often used to extract meaningful information from image data. A vegetation index is a (fractional) combination of two or more spectral bands that separates plant and background data. Due to differences in light intensity and imaged subject, it is often difficult to use absolute values as thresholds to extract plant-relevant data while relative differences are much more stable. Hence, this technique is often used to analyse data from RGB and NIR cameras (or combined versions). VI are also developed for hyperspectral imaging systems (Citation: Vogelmann, Rock& al., 1993 Vogelmann, J., Rock, B. & Moss, D. (1993). Red edge spectral measurements from sugar maple leaves. International Journal of Remote Sensing, 14(8). 1563–1575. https://doi.org/10.1080/01431169308953986 ; Citation: Rouse, & al., 1974 Rouse, J., , & (1974). Monitoring vegetation systems in the Great Plains with ERTS. Retrieved from https://ntrs.nasa.gov/search.jsp?R=19740022614 ; Citation: Gitelson & Merzlyak, 1994 Gitelson, A. & Merzlyak, M. (1994). Spectral Reflectance Changes Associated with Autumn Senescence of Aesculus hippocastanum L. and Acer platanoides L. Leaves. Spectral Features and Relation to Chlorophyll Estimation. Journal of Plant Physiology, 143(3). 286–292. https://doi.org/10.1016/S0176-1617(11)81633-0 ; Citation: Sims & Gamon, 2002 Sims, D. & Gamon, J. (2002). Relationships between leaf pigment content and spectral reflectance across a wide range of species, leaf structures and developmental stages. Remote Sensing of Environment, 81(2). 337–354. https://doi.org/10.1016/S0034-4257(02)00010-X ; Citation: Gamon, Peñuelas& al., 1992 Gamon, J., Peñuelas, J. & Field, C. (1992). A narrow-waveband spectral index that tracks diurnal changes in photosynthetic efficiency. Remote Sensing of Environment, 41(1). 35–44. https://doi.org/10.1016/0034-4257(92)90059-S ; Citation: Behmann, Steinrücken& al., 2014 Behmann, J., Steinrücken, J. & Plümer, L. (2014). Detection of early plant stress responses in hyperspectral images. ISPRS Journal of Photogrammetry and Remote Sensing, 93. 98–111. https://doi.org/10.1016/j.isprsjprs.2014.03.016 ; Citation: Jin & Wang, 2016 Jin, J. & Wang, Q. (2016). Hyperspectral indices based on first derivative spectra closely trace canopy transpiration in a desert plant. Ecological Informatics, 35. 1–8. https://doi.org/10.1016/j.ecoinf.2016.06.004 ; Citation: Gao, Nuyttens& al., 2018 Gao, J., Nuyttens, D., Lootens, P., He, Y. & Pieters, J. (2018). Recognising weeds in a maize crop using a random forest machine-learning algorithm and near-infrared snapshot mosaic hyperspectral imagery. Biosystems Engineering, 170. 39–50. https://doi.org/10.1016/j.biosystemseng.2018.03.006 ; Citation: Alonso, Van Wittenberghe& al., 2017) Alonso, L., Van Wittenberghe, S., Amorós-López, J., Vila-Francés, J., Gómez-Chova, L. & Moreno, J. (2017). Diurnal Cycle Relationships between Passive Fluorescence, PRI and NPQ of Vegetation in a Controlled Stress Experiment. Remote Sensing, 9(8). 770. https://doi.org/10.3390/rs9080770 . These indices are used in linear regression or other machine learning pipelines to calculate an output variable. Due to these observations and wide-spread use in the phenotyping community, we also generated VI. However, only a limited and non-uniformly spaced number of bands are available here, limiting the possibilities to use VI from literature. Therefore, we generated a custom set of VI. Based on literature and taking practical limitations of the number of variables into account, we limited the created VI to combinations of fractions of band pairs, summarised by the equations below. All possible combinations of two spectral bands were generated. \nu^{F}_{ij} is the fraction (\mathit{F}), and \nu^{\mathit{NF}}is the normalised fraction (\mathit{NF}) of a pair of bands. The model automatically selected the relevant indices thanks to regularisation (see below). The spectral bands from H1 were numbered 0 through 24, and those of H2  25 to 40. An index was only included if the absolute value of the maximum value of all Pearson correlation coefficients ( \rho ) with already included indices was lower than 0.95. This boundary ensures that none of the features was (nearly) linearly dependent. Note that the included VI need not be the same for the different data types since the correlation metric might differ. This technique generates up to 2459 new features:

Normally, VI are only generated on image data from plants, but we also generated them for the background materials. Limiting the application of VI to plant data alone would drastically reduce the number of features available for some models. Moreover, the comparison would not be fair since the model based on plant data would have potentially more expressive features than the background model. The data from the different data types is not mixed, so background VI are based solely on background data. The same holds for plant data. A different set of VI were selected for each experiment and data type because of the constraint of the upper value of the correlation coefficient.

To assess the variability in performance due to the subsampling in the second dataset, we constructed nine independent subsets of each subsample size. Consequently, partial overlap between datasets was possible.

A linear model combined with Tikhonov (or L2) regularisation (Citation: Tikhonov, 1963) Tikhonov, A. (1963). On the solution of ill-posed problems and the method of regularization. Doklady Akademii Nauk SSSR, 151. 501–504. Retrieved from http://www.mathnet.ru/php/archive.phtml?wshow=paper&jrnid=dan&paperid=28329&option_lang=eng , commonly called ridge regression, was used to fit the camera data to the environmental and ecophysiological data. Such a model is well-suited to demonstrate the correlation between different datasets and should provide improved prediction performance compared to only using indices (Citation: Yendrek, Tomaz& al., 2017) Yendrek, C., Tomaz, T., Montes, C., Cao, Y., Morse, A., Brown, P., McIntyre, L., Leakey, A. & Ainsworth, E. (2017). High-Throughput Phenotyping of Maize Leaf Physiological and Biochemical Traits Using Hyperspectral Reflectance. Plant Physiology, 173(1). 614–626. https://doi.org/10.1104/pp.16.01447 . It is guaranteed to provide the global optimum for regularised loss with given \lambda and is very fast to fit. This is important since hyperspectral cameras generate vast amounts of data. The whole pipeline was implemented in Python, with the help of the Pandas (Citation: McKinney, 2010 McKinney, W. (2010). Data Structures for Statistical Computing in Python. https://doi.org/10.25080/Majora-92bf1922-00a ; Citation: McKinney, 2011) McKinney, W. (2011). pandas: a Foundational Python Library for Data Analysis and Statistics. Python for High Performance and Scientific Computing, 14. 9. and Scikit-Learn (Citation: Pedregosa, Varoquaux& al., 2011) Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M. & Duchesnay, É. (2011). Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12. 2825−2830. Retrieved from http://jmlr.csail.mit.edu/papers/v12/pedregosa11a.html and publicly available on GitHub.

Ridge regression has one hyperparameter that must be optimised. This parameter determines the impact of the total magnitude of the model coefficients. As a result, each time series had to be split into three subsets: training, validation and test data (as described in Generalisation, Bias and Variance), depicted in figure 5.5. To eliminate possible day-night rhythms, we split the data into batches of 3000 samples (2.5h), while 1500 samples (1.25h) between batches were discarded to eliminate the correlation between adjacent batches. This decorrelation was verified after the analysis by means of an offset between the target and input data (not shown). After selecting the optimal hyperparameter, a final model was trained on both the train and validation data. This ensures optimal data use.

In summary, three different types of linear models were generated based on six different data types. The three model types refer to the kind of input features presented to the model. In the first case, the model was trained on the averaged spectral bands. In the second case, VI were added to the input feature set. This input data expansion enabled the model to leverage nonlinear dependencies in the data. In the third and final case, the input features consisted entirely of spectral bands but without averaging over the entire mask. Different materials give rise to distinct data types; plant 1, plant 2, wood, PVC, Ytong, and cotton were the materials considered in this study. The dataflow through the machine learning pipeline is depicted in figure 5.6.

Additional Details on the Setup #

All variables except for T_\text{air} and h were measured outside the area observed with the camera. This was a requirement due to the high reflectivity of the leaf chamber that resulted in undesired exposure compensation of the camera.

Results #

The data were analysed in three parts: first, the environmental conditions were investigated to better understand the ecophysiological responses in the second and third parts. The averaged dataset was extended with VI and analysed in the second part, and the subsampled dataset was studied in the third and final parts.

Analysis of the Environmental Conditions #

Normalised density plots of the environmental conditions are depicted in figure 5.7. These indicate that I_{\text{PAR}} vs. h (\rho = -0.05) and I_{\text{PAR}} vs. T_\text{air} (\rho = -0.04) did not show bias towards particular values of either parameter. This is, however, not the case for h vs. T_\text{air} (\rho = -0.27), due to the inability of the growth chamber to settle at the target point before a new one was set. A combination of low T_\text{air} and low h was not achieved (see (b), lower left corner). High h values are also less prominent than low ones.

Similar observations were made for the second experiment (not depicted) since the random target sequence was the same for both experiments to simplify the comparison between both extracted datasets. As expected, the main environmental conditions in the growth chamber covered a wide range, but extreme values that cause strong stress conditions were avoided.

Analysis of the Averaged Dataset #

First, we analyse the smaller dataset. This is the dataset that is constructed after averaging over the entire mask, resulting in a single value per spectral band and per mask type. VI were also generated from the spectral bands. These features were then fitted to target ecophysiological and environmental variables. An overview of the model performance for each data type is depicted in figure 5.8. Models with VI are shown by means of coloured bars, while models without VI are depicted using a bar with only a black border.

The performance for photosynthetic rate ( P_n ), stomatal conductance ( g_s ) and transpiration rate ( E ) is poor, both with and without VI. Better performance is observed for T_\text{leaf} and to a lesser extent D_{\text{leaf}} . The entire time series of D_{\text{leaf}} for plant2 is depicted in figure 5.9 for visual inspection. There are several regions where the data fitted very well, but in other parts, there was a significant offset from the measured variable, especially at the extremes. Though the model usually follows the correct trend. Visualisations of different tasks for the plant2 time series are depicted in figure 5.10. From this figure, it is clear that models with a high normalised mean squared error (NMSE) value have very low extraction efficiency. We consider a fit sufficient if the NMSE value is equal to or below 0.25 for both plant models.

While NMSE provides an objective comparison, correctly followed trends such as in the case of P_n in figure 5.10 are not reflected in a low NMSE score. This is due to a continued offset between the correctly predicted trends and the measured variable. The prediction of P_n has a better NMSE score because the predicted values approximate an averaged version of the measured variable.

Ecophysiological tasks with high NMSE values also display major differences in modelling performance between batches of test data (not visualised). This further indicates that the model was unable to extract meaningful information from the spectral features and VI. Either the information was masked by interferers, or the camera was unable to capture changes related to these variables.

For most tasks, there was a limited improvement or even reduced performance by adding VI. Improved performance is strongest for D_{\text{leaf}} , T_\text{leaf} , and T_\text{air} . Reduced performance on the test data can be attributed to overfitting of the model. The performance on the training set has improved, but the models’ ability to generalise has decreased, leading to reduced performance. This is especially visible in the case of h when using Ytong-data. The NMSE value rises from \num{0.63} to \num{1.13}, above the baseline.

For all physiological variables, plant1 and plant2 are expected to have better performance than the background materials. However, this is not the case for plant2. While the differences between plant2 and Yong and cotton (plant and materials from the second experiment) are low, there is a large difference between plant1 and plant2 for both the average modelling performance of P_n , g_s and E .

From the environmental variables, T_\text{air} and I_{\text{PAR}} have the highest extraction precision. h has a similar NMSE score for all data types except for Ytong.

Analysis of the Subsampling Dataset #

The averaged dataset loses the differences in response due to for instance the age of a leaf. Including such dynamics can possibly yield improved performance. This subsampled dataset investigates this by means of including randomly sampled pixels. The ridge model is able to automatically differentiate between more and less relevant pixels because of the regularisation. The analysis itself was similar to that of the previous section.

Figure 5.11 displays the performance for the same set of tasks as figure 5.7. Error bars are depicted in black. They were computed from the difference in NMSE value arising from the subsampling. The performance of the tasks is similar to that of the previous dataset without VI. These are not included here because it would drastically increase the number of features, which is undesirable.

The number of samples is an important metric for this dataset. Figure 5.12 visualises the effect of the number of features on the modelling accuracy for T_\text{air} and E . The standard deviation is depicted as a shaded area around the curve in the same colour. As expected, the NMSE decreases with an increasing number of features. The largest subsample size generally offers the best performance, but for underperforming tasks such as E , the variance between different batches tends to increase with an increasing number of samples, especially for plant data. This further illustrates the lack of useful information in the dataset for this particular task.

Discussion #

In this chapter, we set up two experiments to demonstrate PRC with plants using a hyperspectral sensor. However, results from figure 5.7, figure 5.11 indicate that there is no clear difference in task performance between data used from plants and background materials. In most cases, the NMSE values are very similar, both with and without VI. This indicates that the plant’s state dynamics are not captured and could not be used for PRC. We also compared these results with other studies that investigated how hyperspectral cameras can be used to estimate ecophysiological parameters.

The only ecophysiological variables with good performance for both plant time series are D_{\text{leaf}} and T_\text{leaf} . However, the background is also good at predicting these. This can be explained by the high correlation between T_\text{air} and the variables D_{\text{leaf}} and T_\text{leaf} , depicted in table 5.1. D_{\text{leaf}} is driven by relative humidity and leaf temperature, which are in turn affected by air temperature, light intensity and water status (Citation: Amitrano, Arena& al., 2019) Amitrano, C., Arena, C., Rouphael, Y., De Pascale, S. & De Micco, V. (2019). Vapour pressure deficit: The hidden driver behind plant morphofunctional traits in controlled environments. Annals of Applied Biology, 175(3). 313–325. https://doi.org/10.1111/aab.12544 . The reflection spectrum of plants in the near-infrared region varies when the air temperature changes (Citation: Carter, Bahadur& al., 2000) Carter, G., Bahadur, R. & Norby, R. (2000). Effects of elevated atmospheric CO2 and temperature on leaf optical properties in Acer saccharum. Environmental and Experimental Botany, 43(3). 267–273. https://doi.org/10.1016/S0098-8472(00)00048-4 . These changes are probably detected by the model, leading to the good performance of temperature-related variables. The backgrounds considered here might have similar reflective properties. Though their structure cannot change as in the case of plants, they can radiate more or less infrared light, enabling the model to extract the temperature (Citation: Beltrán, García& al., 1997) Beltrán, M., García, J. & Marcilla, A. (1997). Infrared spectral changes in PVC and plasticized PVC during gelation and fusion. European Polymer Journal, 33(4). 453–462. https://doi.org/10.1016/S0014-3057(96)00213-3 . The spectrum at two different temperatures is depicted in figure 5.14. The spectra of both materials and the plant have clearly changed in the NIR region.

The results for g_s are in line with previous research by Citation: Zarco-Tejada, Morales& al. (2013) Zarco-Tejada, P., Morales, A., Testi, L. & Villalobos, F. (2013). Spatio-temporal patterns of chlorophyll fluorescence and physiological and structural indices acquired from hyperspectral imagery as compared with carbon fluxes measured with eddy covariance. Remote Sensing of Environment, 133. 102–115. https://doi.org/10.1016/j.rse.2013.02.003 , who used a similar wavelength range from 400 to 900nm and were also unable to extract this parameter. However, others have reported successful extraction of g_s . Citation: Rapaport, Hochberg& al. (2015) Rapaport, T., Hochberg, U., Shoshany, M., Karnieli, A. & Rachmilevitch, S. (2015). Combining leaf physiology, hyperspectral imaging and partial least squares-regression (PLS-R) for grapevine water status assessment. ISPRS Journal of Photogrammetry and Remote Sensing, 109. 88–97. https://doi.org/10.1016/j.isprsjprs.2015.09.003 were able to assess g_s in grapevine, using the WABI-3 VI. This index uses a spectral band at 1485nm, which indicates that a wider spectral range is necessary to assess g_s .

The possibility to estimate P_n from hyperspectral data in maize has been reported by Citation: Yendrek, Tomaz& al. (2017) Yendrek, C., Tomaz, T., Montes, C., Cao, Y., Morse, A., Brown, P., McIntyre, L., Leakey, A. & Ainsworth, E. (2017). High-Throughput Phenotyping of Maize Leaf Physiological and Biochemical Traits Using Hyperspectral Reflectance. Plant Physiology, 173(1). 614–626. https://doi.org/10.1104/pp.16.01447 . They were able to predict the carbon dioxide (CO2)-saturated rate of photosynthesis (V_\text{max}) from hyperspectral imaging and a partial least squares (PLS) regression model. They indicated important peak wavelengths at 554nm and 719nm. Though the VIS and NIR camera have nearby bands at 552nm and 724nm, there are fewer bands available compared to the study by Citation: Yendrek, Tomaz& al. (2017) Yendrek, C., Tomaz, T., Montes, C., Cao, Y., Morse, A., Brown, P., McIntyre, L., Leakey, A. & Ainsworth, E. (2017). High-Throughput Phenotyping of Maize Leaf Physiological and Biochemical Traits Using Hyperspectral Reflectance. Plant Physiology, 173(1). 614–626. https://doi.org/10.1104/pp.16.01447 , which acquired the spectral reflection of maize at a resolution of 1nm.

As mentioned in the introduction, sun-induced fluorescence (SIF) is a promising indicator for photosynthetic activity. It originates from initial reactions in Photosystem II and I, which have narrow peaks around 690nm and 760nm respectively. Resolving these peaks requires high spectral resolution (\le 5nm), which is not possible with the technology used in this study (Citation: Bandopadhyay, Rastogi& al., 2020) Bandopadhyay, S., Rastogi, A. & Juszczak, R. (2020). Review of Top-of-Canopy Sun-Induced Fluorescence (SIF) Studies from Ground, UAV, Airborne to Spaceborne Observations. Sensors, 20(4). 1144. https://doi.org/10.3390/s20041144 . A snapshot camera with improved resolution should be able to compute SIF and thus be able to better assess photosynthetic activity.

Unlike the current study, Citation: Marshall, Thenkabail& al. (2016) Marshall, M., Thenkabail, P., Biggs, T. & Post, K. (2016). Hyperspectral narrowband and multispectral broadband indices for remote sensing of crop evapotranspiration and its components (transpiration and soil evaporation). Agricultural and Forest Meteorology, 218-219. 122–134. https://doi.org/10.1016/j.agrformet.2015.12.025 were able to measure E with hyperspectral imagery in cotton, rice and maize. They used a spectral range from 428 to 2295nm with a resolution between 1nm and 10nm, which was wavelength dependent. Their most important VI was HNDVI (Citation: Gao, 1996) Gao, B. (1996). NDWI—A normalized difference water index for remote sensing of vegetation liquid water from space. Remote Sensing of Environment, 58(3). 257–266. https://doi.org/10.1016/S0034-4257(96)00067-3 , which uses a band at 845nm and 1256nm.

From these studies, we identified three causes that might explain the poor results for g_s , P_n , and E . First, the spectral range was insufficient. More specifically, higher wavelengths up to at least 1500nm are needed to assess these properties based on literature. Second, the spectral resolution needs to increase to preferably 1nm, and the band spacing should be more uniform. Third, other studies which investigated ecophysiological properties applied clear treatments (Citation: Zhao, Reddy& al., 2005 Zhao, D., Reddy, K., Kakani, V. & Reddy, V. (2005). Nitrogen deficiency effects on plant growth, leaf photosynthesis, and hyperspectral reflectance properties of sorghum. European Journal of Agronomy, 22(4). 391–403. https://doi.org/10.1016/j.eja.2004.06.005 ; Citation: Zarco-Tejada, Morales& al., 2013 Zarco-Tejada, P., Morales, A., Testi, L. & Villalobos, F. (2013). Spatio-temporal patterns of chlorophyll fluorescence and physiological and structural indices acquired from hyperspectral imagery as compared with carbon fluxes measured with eddy covariance. Remote Sensing of Environment, 133. 102–115. https://doi.org/10.1016/j.rse.2013.02.003 ; Citation: Behmann, Steinrücken& al., 2014 Behmann, J., Steinrücken, J. & Plümer, L. (2014). Detection of early plant stress responses in hyperspectral images. ISPRS Journal of Photogrammetry and Remote Sensing, 93. 98–111. https://doi.org/10.1016/j.isprsjprs.2014.03.016 ; Citation: Silva-Perez, Molero& al., 2018) Silva-Perez, V., Molero, G., Serbin, S., Condon, A., Reynolds, M., Furbank, R. & Evans, J. (2018). Hyperspectral reflectance as a tool to measure biochemical and physiological traits in wheat. Journal of Experimental Botany, 69(3). 483–496. https://doi.org/10.1093/jxb/erx421 . These cause more pronounced spectral and ecophysiological changes. Therefore, estimation is more direct and less prone to disturbances that mask ecophysiological changes.

Both I_{\text{PAR}} and T_\text{air} are well assessed for plants and all backgrounds. The good performance of I_{\text{PAR}} is expected since the camera directly observes light and can reconstruct the spectrum used in the measurement of I_{\text{PAR}} . As mentioned before, spectral changes due to temperature can explain the good modelling performance of T_\text{air} . It was verified that the camera’s responses are not significantly temperature-dependent other than increased white noise at elevated temperatures (data not presented).

The poor NMSE value of h from the data is in stark contrast to the other environmental variables. A possible reason why these have a high NMSE value is that there is no infrared radiation around 2µm captured by the camera nor emitted by the lights. Consequently, none of the strong absorption peaks of water was captured.

Conclusion #

We investigated the use of hyperspectral cameras for PRC with plants. Based on the results obtained, we can conclude that hyperspectral cameras are not suited to capture the response dynamics of plants. However, these results are still valuable for the plant science community. The novelty of this study is that ecophysiological parameters were investigated without the presence of strong biotic or abiotic stress factors with high temporal resolution (3s) over the duration of one week. Increasingly, researchers are investigating the dynamic behaviour of plants as these are crucial in determining plant productivity in continuously fluctuating environments such as those in agricultural fields. The analysis indicated that the information content of the hyperspectral data is low for P_n , g_s , and E . A possible explanation is limited variability due to the much lower treatment effect and low-resolution spectral sensing capabilities. The lack of a clear stress-inducing treatment causes less variation of the ecophysiological variables observed. Induced variations were possibly not significant enough to be detectable among other interfering and noise signals that are present in the image data. Additionally, the spectral resolution and range were limited. D_{\text{leaf}} and T_\text{leaf} are good performing ecophysiological task (i.e. have low NMSE scores). Environmental variables also show varying results. As expected, I_{\text{PAR}} is well assessed, as is T_\text{air} . h cannot be extracted due to the lack of wavelengths above 2µm. We suspect that the reflection spectrum changes depending on temperature, which enables the model to accurately predict T_\text{air} . T_\text{leaf} and D_{\text{leaf}} are strongly correlated to T_\text{air} might explain why these ecophysiological parameters could be assessed. In summary, current snapshot hyperspectral technologies are not yet well suited to monitor the dynamic responses of plants. Major improvements upon sensitivity and spectral resolution are probably required to enable the detection of subtle changes of ecophysiological parameters in stress-free conditions.

Supplementary Information #