Geoinformatics Unit

Wei HE


Current Position

Wei He is currently a research scientist at Geoinformatics Unit, the RIKEN Center for Advanced Intelligence Project (AIP), Japan. His research interests include remote sensing image restoration, low-rank modeling, and deep learning.
He is a member of the IEEE and IEEE Geoscience and Remote Sensing Society (GRSS).


2020 Jul - Present    Research Scientist, RIKEN AIP, Japan
2018 Jan - 2020 Jun    Postdoctoral Researcher, RIKEN AIP, Japan
2012 Sep - 2017 Jun    Ph.D. LIESMARS, Wuhan University, Wuhan, China

Journal Papers

  1. D. Hong, W. He, N. Yokoya, J. Yao, L. Gao, L. Zhang, J. Chanussot, and X.X. Zhu, " Interpretable hyperspectral AI: When non-convex modeling meets hyperspectral remote sensing ," IEEE Geoscience and Remote Sensing Magazine, 2021.
    PDF    Quick Abstract

    Abstract: Hyperspectral imaging, also known as image spectrometry, is a landmark technique in geoscience and remote sensing (RS). In the past decade, enormous efforts have been made to process and analyze these hyperspectral (HS) products mainly by means of seasoned experts. However, with the ever-growing volume of data, the bulk of costs in manpower and material resources poses new challenges on reducing the burden of manual labor and improving efficiency. For this reason, it is, therefore, urgent to develop more intelligent and automatic approaches for various HS RS applications. Machine learning (ML) tools with convex optimization have successfully undertaken the tasks of numerous artificial intelligence (AI)-related applications. However, their ability in handling complex practical problems remains limited, particularly for HS data, due to the effects of various spectral variabilities in the process of HS imaging and the complexity and redundancy of higher dimensional HS signals. Compared to the convex models, non-convex modeling, which is capable of characterizing more complex real scenes and providing the model interpretability technically and theoretically, has been proven to be a feasible solution to reduce the gap between challenging HS vision tasks and currently advanced intelligent data processing models. This article mainly presents an advanced and cutting-edge technical survey for non-convex modeling towards interpretable AI models covering a board scope in the following topics of HS RS: 1) HS image restoration, 2) dimensionality reduction, 3) data fusion and enhancement, 3) spectral unmixing, 4) cross-modality learning for large-scale land cover mapping. Around these topics, we will showcase the significance of non-convex techniques to bridge the gap between HS RS and interpretable AI models with a brief introduction on the research background and motivation, an emphasis on the resulting methodological foundations and solution, and an intuitive clarification of illustrative examples. At the end of each topic, we also pose the remaining challenges on how to completely model the issues of complex spectral vision from the perspective of intelligent ML combined with physical priors and numerical non-convex modeling, and accordingly point out future research directions. This paper aims to create a good entry point to the advanced literature for experienced researchers, Ph.D. students, and engineers who already have some background knowledge in HS RS, ML, and optimization. This can further help them launch new investigations on the basis of the above topics and interpretable AI techniques for their focused fields.

  2. N. Yokoya, K. Yamanoi, W. He, G. Baier, B. Adriano, H. Miura, and S. Oishi, " Breaking limits of remote sensing by deep learning from simulated data for flood and debris flow mapping ," IEEE Transactions on Geoscience and Remote Sensing (Early Access), 2021.
    PDF    Quick Abstract

    Abstract: We propose a framework that estimates the inundation depth (maximum water level) and debris-flow-induced topographic deformation from remote sensing imagery by integrating deep learning and numerical simulation. A water and debris-flow simulator generates training data for various artificial disaster scenarios. We show that regression models based on Attention U-Net and LinkNet architectures trained on such synthetic data can predict the maximum water level and topographic deformation from a remote sensing-derived change detection map and a digital elevation model. The proposed framework has an inpainting capability, thus mitigating the false negatives that are inevitable in remote sensing image analysis. Our framework breaks limits of remote sensing and enables rapid estimation of inundation depth and topographic deformation, essential information for emergency response, including rescue and relief activities. We conduct experiments with both synthetic and real data for two disaster events that caused simultaneous flooding and debris flows and demonstrate the effectiveness of our approach quantitatively and qualitatively. Our code and data sets are available at

  3. W. He, Q. Yao, C. Li, N. Yokoya, Q. Zhao, H. Zhang, and L. Zhang, " Non-local meets global: An iterative paradigm for hyperspectral image restoration ," IEEE Transactions on Pattern Analysis and Machine Intelligence (early access), 2020.
    Quick Abstract

    Abstract: Non-local low-rank tensor approximation has been developed as a state-of-the-art method for hyperspectral image (HSI) restoration, which includes the tasks of denoising, compressed HSI reconstruction and inpainting. Unfortunately, while its restoration performance benefits from more spectral bands, its runtime also substantially increases. In this paper, we claim that the HSI lies in a global spectral low-rank subspace, and the spectral subspaces of each full band patch group should lie in this global low-rank subspace. This motivates us to propose a unified paradigm combining the spatial and spectral properties for HSI restoration. The proposed paradigm enjoys performance superiority from the non-local spatial denoising and light computation complexity from the low-rank orthogonal basis exploration. An efficient alternating minimization algorithm with rank adaptation is developed. It is done by first solving a fidelity term-related problem for the update of a latent input image, and then learning a low-dimensional orthogonal basis and the related reduced image from the latent input image. Subsequently, non-local low-rank denoising is developed to refine the reduced image and orthogonal basis iteratively. Finally, the experiments on HSI denoising, compressed reconstruction, and inpainting tasks, with both simulated and real datasets, demonstrate its superiority with respect to state-of-the-art HSI restoration methods.

  4. Y. Chen, T.-Z. Huang, W. He, N. Yokoya, and X.-L. Zhao, " Hyperspectral Image Compressive Sensing Reconstruction Using Subspace-based Nonlocal Tensor Ring Decomposition ," IEEE Transactions on Image Processing (Early Access), pp. 1-16, 2020.
    Quick Abstract

    Abstract: Hyperspectral image compressive sensing reconstruction (HSI-CSR) can largely reduce the high expense and low efficiency of transmitting HSI to ground stations by storing a few compressive measurements, but how to precisely reconstruct the HSI from a few compressive measurements is a challenging issue. It has been proven that considering the global spectral correlation, spatial structure, and nonlocal selfsimilarity priors of HSI can achieve satisfactory reconstruction performances. However, most of the existing methods cannot simultaneously capture the mentioned priors and directly design the regularization term to the HSI. In this article, we propose a novel subspace-based nonlocal tensor ring decomposition method (SNLTR) for HSI-CSR. Instead of designing the regularization of the low-rank approximation to the HSI, we assume that the HSI lies in a low-dimensional subspace. Moreover, to explore the nonlocal self-similarity and preserve the spatial structure of HSI, we introduce a nonlocal tensor ring decomposition strategy to constrain the related coefficient image, which can decrease the computational cost compared to the methods that directly employ the nonlocal regularization to HSI. Finally, a well-known alternating minimization method is designed to efficiently solve the proposed SNLTR. Extensive experimental results demonstrate that our SNLTR method can significantly outperform existing approaches for HSI-CSR.

  5. G. Baier, W. He, and N. Yokoya, " Robust nonlocal low-rank SAR time series despeckling considering speckle correlation by total variation regularization ," IEEE Transactions on Geoscience and Remote Sensing (Early Access), 2020.
    Quick Abstract

    Abstract: Outliers and speckle both corrupt synthetic aperture radar (SAR) time series. Furthermore, due to the coherence between SAR acquisitions, their speckle can no longer be regarded as independent. We propose a nonlocal low-rank time series despeckling algorithm that is robust against outliers and also specifically addresses speckle correlation between acquisitions. By imposing a total variation regularization on the signal’s speckle component, its correlation between acquisition can be captured, facilitating the extraction of outliers from the unfiltered signal and correlated speckle. Robustness against outliers also addresses matching errors and inaccuracies in the nonlocal similarity search. Such errors include mismatched data in the nonlocal estimation process, which degrade denoising performance in conventional similarity-based filtering approaches. Multiple experiments on real and synthetic data assess the proposed approaches performance by comparing it to state-of-the-art methods. It provides filtering results of comparable quality but is not adversely affected by outliers.

  6. T. Uezato, N. Yokoya, and W. He, " Illumination invariant hyperspectral image unmixing based on a digital surface model ," IEEE Transactions on Image Processing, vol. 29, no. 1, pp. 3652-3664, 2019.
    Quick Abstract

    Abstract: Although many spectral unmixing models have been developed to address spectral variability caused by variable incident illuminations, the mechanism of the spectral variability is still unclear. This paper proposes an unmixing model, named illumination invariant spectral unmixing (IISU). IISU makes the first attempt to use the radiance hyperspectral data and a LiDAR-derived digital surface model (DSM) in order to physically explain variable illuminations and shadows in the unmixing framework. Incident angles, sky factors, visibility from the sun derived from the LiDAR-derived DSM support the explicit explanation of endmember variability in the unmixing process from radiance perspective. The proposed model was efficiently solved by a straightforward optimization procedure. The unmixing results showed that the other state-of-the-art unmixing models did not work well especially in the shaded pixels. On the other hand, the proposed model estimated more accurate abundances and shadow compensated reflectance than the existing models.

  7. Y. Chen, W. He, N. Yokoya, and T.-Z. Huang, " Non-local tensor ring decomposition for hyperspectral image denoising ," IEEE Trans. Geosci. Remote Sens., vol. 58, no. 2, pp. 1348-1362, 2019.
    Quick Abstract

    Abstract: Hyperspectral image (HSI) denoising is a fundamental problem in remote sensing and image processing. Recently, non-local low-rank tensor approximation based denoising methods have attracted much attention, due to the advantage of fully exploiting the non-local self-similarity and global spectral correlation. Existing non-local low-rank tensor approximation methods were mainly based on two common Tucker or CP decomposition and achieved the state-of-the-art results, but they suffer some troubles and are not the best approximation for a tensor. For example, the number of parameters of Tucker decomposition increases exponentially follow its dimension, and CP decomposition cannot better preserve the intrinsic correlation of HSI. In this paper, we propose a non-local tensor ring (TR) approximation for HSI denoising by utilizing TR decomposition to simultaneously explore non-local self-similarity and global spectral low-rank characteristic. TR decomposition approximates a high-order tensor as a sequence of cyclically contracted three-order tensors, which has a strong ability to explore these two intrinsic priors and improve the HSI denoising result. Moreover, we develop an efficient proximal alternating minimization algorithm to efficiently optimize the proposed TR decomposition model. Extensive experiments on three simulated datasets under several noise levels and two real datasets testify that the proposed TR model performs better HSI denoising results than several state-of-the-art methods in term of quantitative and visual performance evaluations.

  8. Y. Chen, W. He, N. Yokoya, and T.-Z. Huang, " Blind cloud and cloud shadow removal of multitemporal images based on total variation regularized low-rank sparsity decomposition ," ISPRS Journal of Photogrammetry and Remote Sensing, vol. 157, pp. 93-107, 2019.
    Quick Abstract

    Abstract: Cloud and cloud shadow (cloud/shadow) removal from multitemporal satellite images is a challenging task and has elicited much attention for subsequent information extraction. Regarding cloud/shadow areas as missing information, low-rank matrix/tensor completion based methods are popular to recover information undergoing cloud/shadow degradation. However, existing methods required to determine the cloud/shadow locations in advance and failed to completely use the latent information in cloud/shadow areas. In this study, we propose a blind cloud/shadow removal method for time-series remote sensing images by unifying cloud/shadow detection and removal together. First, we decompose the degraded image into low-rank clean image (surface-reflected) component and sparse (cloud/shadow) component, which can simultaneously and completely use the underlying characteristics of these two components. Meanwhile, the spatial-spectral total variation regularization is introduced to promote the spatial-spectral continuity of the cloud/shadow component. Second, the cloud/shadow locations are detected from the sparse component using a threshold method. Finally, we adopt the cloud/shadow detection results to guide the information compensation from the original observed images to better preserve the information in cloud/shadow-free locations. The problem of the proposed model is efficiently addressed using the alternating direction method of multipliers. Both simulated and real datasets are performed to demonstrate the effectiveness of our method for cloud/shadow detection and removal when compared with other state-of-the-art methods.

  9. Y. Chen, W. He, N. Yokoya, and T.-Z. Huang, " Hyperspectral image restoration using weighted group sparsity regularized low-rank tensor decomposition ," IEEE Transactions on Cybernetics (accepted for publication), 2019.
    Quick Abstract

    Abstract: Mixed noise (such as Gaussian, impulse, stripe, and deadline noises) contamination is a common phenomenon in hyperspectral imagery (HSI), greatly degrading visual quality and affecting subsequent processing accuracy. By encoding sparse prior to the spatial or spectral difference images, total variation (TV) regularization is an efficient tool for removing the noises. However, the previous TV term cannot maintain the shared group sparsity pattern of the spatial difference images of different spectral bands. To address this issue, this study proposes a group sparsity regularization of the spatial difference images for HSI restoration. Instead of using L1 or L2-norm (sparsity) on the difference image itself, we introduce a weighted L2,1-norm to constrain the spatial difference image cube, efficiently exploring the shared group sparse pattern. Moreover, we employ the well-known low-rank Tucker decomposition to capture the global spatial-spectral correlation from three HSI dimensions. To summarize, a weighted group sparsity regularized low-rank tensor decomposition (LRTDGS) method is presented for HSI restoration. An efficient augmented Lagrange multiplier algorithm is employed to solve the LRTDGS model. The superiority of this method for HSI restoration is demonstrated by a series of experimental results from both simulated and real data, as compared to other state-of-the-art TV regularized low-rank matrix/tensor decomposition methods.

  10. W. He, N. Yokoya, L. Yuan, and Q. Zhao, " Remote sensing image reconstruction using tensor ring completion and total-variation ," IEEE Trans. Geosci. Remote Sens., vol. 57, no. 11, pp. 8998-9009, 2019.
    Quick Abstract

    Abstract: Time-series remote sensing (RS) images are often corrupted by various types of missing information such as dead pixels, clouds, and cloud shadows that significantly influence the subsequent applications. In this paper, we introduce a new low-rank tensor decomposition model, termed tensor ring (TR) decomposition, to the analysis of RS datasets and propose a TR completion method for the missing information reconstruction. The proposed TR completion model has the ability to utilize the low-rank property of time-series RS images from different dimensions. To furtherly explore the smoothness of the RS image spatial information, total-variation regularization is also incorporated into the TR completion model. The proposed model is efficiently solved using two algorithms, the augmented Lagrange multiplier (ALM) and the alternating least square (ALS) methods. The simulated and real data experiments show superior performance compared to other state-of-the-art low-rank related algorithms.

  11. W. He and N. Yokoya, " Multi-temporal Sentinel-1 and -2 data fusion for optical image simulation ," ISPRS International Journal of Geo-Information, vol. 7, no. 10, pp. 389, 2018.
    PDF    Quick Abstract

    Abstract: In this paper, we present the optical image simulation from synthetic aperture radar (SAR) data using deep learning based methods. Two models, i.e., optical image simulation directly from the SAR data and from multi-temporal SAR-optical data, are proposed to testify the possibilities. The deep learning based methods that we chose to achieve the models are a convolutional neural network (CNN) with a residual architecture and a conditional generative adversarial network (cGAN). We validate our models using the Sentinel-1 and -2 datasets. The experiments demonstrate that the model with multi-temporal SAR-optical data can successfully simulate the optical image, meanwhile, the model with simple SAR data as input failed. The optical image simulation results indicate the possibility of SAR-optical information blending for the subsequent applications such as large-scale cloud removal, and optical data temporal super-resolution. We also investigate the sensitivity of the proposed models against the training samples, and reveal possible future directions.

Conference Papers

  1. T. Uezato, D. Hong, N. Yokoya, W. He, "Guided deep decoder: Unsupervised image pair fusion," European Conference on Computer Vision (ECCV) (spotlight), 2020.
  2. W. He, Q. Yao, C. Li, N. Yokoya, and Q. Zhao, "Non-local meets global: An integrated paradigm for hyperspectral denoising," Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  3. C. Li, W. He, L. Yuan, Z. Sun, and Q. Zhao, "Guaranteed matrix completion under multiple linear transformations," Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  4. W. He, L. Yuan, and N. Yokoya, "Total-variation-regularized tensor ring completion for remote sensing image reconstruction," International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2019.
  5. H. Xu, H. Zhang, W. He, and L. Zhang, "Superpixel based dimension reduction for hyperspectral imagery," IEEE International Geoscience and Remote Sensing Symposium, 2018.