Development of Image Inpainting for object removal using Enhanced Patch Priority and Matching Measures

: Image inpainting can be used to fix broken images and get rid of distracting elements. In exemplar based methods, patch priority computation and exemplar patch selection are crucial to the success of image inpainting technique. The dropping effect occurred in the highest patch priority computation and matching error in the best patch selection are the major issues in the exemplar inpaint methods. In this paper, the enhanced priority calculation technique is employed to avoid the dropping effect and introduced the new similarity measuring process, Mean Squared Error (MSD). The efficacy of the proposed techniques is estimated by comparing with the available methods in the literature qualitatively. It shows that proposed methods outperformed existing techniques.


Introduction
The practice of image inpainting evolved from attempts to repair previously created artworks.The concept behind it is to apply effective information that was not affected by the harm to restore the affected areas in accordance with predetermined guidelines.Its primary goal is to ensure that viewers of the restored image (who are not familiar with the original) are unable to detect any evidence of the restoration process.Scratches refurbishment of old photographs and valuable literatures, robot vision, film industry and television special effects production are just some of the many applications of inpainting technology, which has exploded in popularity thanks to the rapid expansion of computer and multimedia technology.The classification of image inpainting methods encompasses diffusion-based, texture synthesis-based, and exemplar-based approaches.Diffusion-based inpainting techniques are very suitable for the removal of text, scratches, noise, and the restoration of deleted blocks in transmitted images.
The aforementioned approaches exhibit the presence of the staircase effect, contrast loss, and edge blur in the inpainted outcomes, rendering them unsuitable for the purpose of inpainting huge holes.Texture synthesis technologies are employed to restore images that include significant amounts of texture information.The aforementioned methods exhibit visual artifacts along their peripheries.Exemplar-based inpainting techniques are particularly well-suited for images that exhibit regular or organized textures.These techniques yield visually impeccable outcomes, especially when dealing with extensive inpainting areas.The effectiveness of exemplar-based approaches primarily relies on the calculation of the most prioritized patch within the perimeter of the target region and the optimal patch selection from the source region.
1.1 Literature Survey: Image Inpainting PDE-based approaches are utilized to solve partial differential Equations, facilitating the smooth propagation of effective information towards the dented region along the isophote direction.In their study, Rathish et al. [1] incorporated the regularization term by utilizing the square of the L2 norm of the Hessian of the image.They employed convexity separation techniques to solve the resultant semi-discrete method in the Fourier domain.In their study, Yang et al. [2] employed the recently introduced fractional-order structure tensor as a means to govern the regularization procedure.The novel model has the capability to incorporate the authentic anisotropy of tensor regularization, hence enhancing its ability to effectively capture intricate details and intricate structures.Theljani et al. [3] employed a 4 th -order variational model and implemented an adaptive approach to determine diffusion parameters.This was done to enhance the regularization effects specifically in the vicinity of tiny features.In their study, Mousavi et al. [4] examined the impact of the magnitude and phase of the Fourier transform on image restoration.They proposed the use of two regularization parameters and incorporated two degrees of freedom in their approach.In their study, Liu et al. [5] employed the Structural Similarity Index (SSIM) to identify the most comparable exemplar patch.This was achieved by considering four scenarios of rotation and inversion to generate the best candidate patch.The influential study conducted by Isogawa et al. [6] highlighted the significant impact of the mask on the outcomes of restoration.The researchers put forth a mask optimization technique and successfully employed it to achieve favorable results in an automated manner.In their study, Liu et al. [7] made modifications to the confidence term by transforming it into an exponential form.They then proceeded to calculate the addition of confidence term and data term in order to enhance the rationality of the filling direction.In their study, Zhang et al. [8] employed the utilization of curvature and gradient information as a substitute for the data term in order to enhance the filling order.Nevertheless, the researchers did not make any enhancements to the matching method, perhaps resulting in a mismatch mistake occurring between patches during the inpainting procedure.In their study, Nan et al. [9] employed the golden section to assign distinct weights to the data item and confidence item.This approach aimed to enhance the rationality of the restoration order.However, it was found to be ineffective in preventing the occurrence of mismatch errors, indicating a need for improvement in the restoration effect.In their study, Yao [10] incorporated the similarity between the target patch and the surrounding patch into the computation of priority.Additionally, they made a modification by replacing multiplication with addition.Furthermore, she devised a novel similarity calculation function with the aim of enhancing the restoration efficacy.
In their study, Ghorai et al. [11] put forth a image inpainting technique that relies on a Markov Random Field (MRF) basis.The researchers employed a unique approach to group formation, utilizing subspace clustering, in order to selectively explore potential patches inside the relevant source region.Additionally, they implemented a patch refining scheme that leveraged higher order Singular Value Decomposition (SVD) to effectively catch the underlying patterns present among the candidate patches.In their study, Zhang et al. [12] employed surface fitting as a form of aforementioned knowledge and incorporated the Jaccard similarity coefficient to enhance the precision of patch matching.The authors Janardhana Rao et al. [13][14][15] introduced an improved approach for priority computation, which involves the integration of regularization factors and adaptive coefficients.Zhang et al. [16] employed the combination of mean squared diffrerence and square of mean differences as a similarity metric to find the exemplar patch.The video inpainting using exemplar based methods are also evevated in recent years [17][18][19][20][21][22][23][24].This study aimed to determine highest priority patch enhanced method to avoid dropping effect and new patch selection proces that yield favorable inpainting outcomes.The attention of numerous scholars has been drawn to these exemplar-based methodologies, leading to the continual proposal of various enhanced techniques.
The remaining portions of the study are summarized as follows: Section 2 presents the problems identified in current image inpainting methods.Section 3 presents the proposed framework of the image inpainting.The experimental results and the corresponding discussions are described in Section 4. The paper is concluded in section 5.

Exemplar based Inpainting Technique
The process of inpainting in exemplar-based image inpainting methods relies on the execution of two fundamental procedures.Initially, the task involves the identification of the patch with the highest priority located on the perimeter of the target region.Subsequently, an exploration is conducted to identify the most suitable patch from the source region that closely resembles the highest priority patch in the destination region.The comprehensive depiction of the exemplar-based inpainting technique is presented in Figure 1.This technique was initially introduced by Criminisi et al. in their seminal work [25].The designated image to undergo inpainting involves a specific area known as the target region (Ω), which requires restoration, while the rest of the image is referred to as the source region (Φ).Several patches were generated along the perimeter of the target region (∂Ω), with the pixels located on the boundary serving as the central pixel of each patch (Ψp).The patch with the highest priority among numerous patches located on the boundary of the target region was determined by calculating a priority function.The priority of the patch centered at pixel p was determined.
� =  � *  � (1) where,  � denotes the confidence term and  � represents the data term.Subsequently, the most significant patch located at the boundary of the target region was populated with the most optimal identical patches �Ψ � � , Ψ � �� � generated within the source region.These patches are commonly referred to as exemplars.The identification of the most comparable patch from the patches generated in the source region is accomplished through the utilization of the Sum of Squared Difference (SSD) distance metric, which quantifies the dissimilarity between the highest priority patch and the patches inside the source region.The patch exhibiting the lowest sum of squared differences (SSD) is considered the most suitable candidate inside the source region for filling the patch with the highest priority.This evaluation was performed using equation (2). where, where,  � � ,  � � and  � � are red, green, and blue planes respectively in target region. � � � ,  � � � and  � � � are red, green, and blue planes respectively in source region.
Subsequently, the designated patch �Ψ � � was populated with the most optimal exemplar patch Ψ � � .The boundary of the target zone was scheduled to be updated, and the entire procedure was iterated until the target region in the image was fully completed.

Problem Statement
In this paper, it is concentrated on object removal and filling the hole after reving the object using exemplar based image inpainting technique.The parameters represented in this techniques are shown figure 2. Φ is the source region, Ω gives the region formed after removing the unwanted object in the image called target region.Ω indicates the border of Ω.The patches on the boundary are represented with Ψ �� , Ψ �� , Ψ �� , Ψ �� which are shown in figure 3. The patches on the source region are Ψ �� , Ψ �� , Ψ �� shown in figure 3. The exemplar-based inpainting technique has two primary processes.The first step entails the identification of the most prioritized patch located on the boundary of the target region.The second step involves the identification of the most similar patch, in terms of the highest priority patch, within the source region.
In both the stages it is identified significant problems in the existing techniques.There exist a dropping effect while identifying the highest priority patch and mismatching error occurs during the similar patch detection in the source region.During the calculation of priority value of the patches on the boundary, this value reduces to a very lower value for less number of iterations in the process, it is called the dropping effect.After identifying thepatch with highest priority, it is filled by selecting the most similar patch from the source region.The suitable similarity measure gives the best inpainting results.It may cause mismatch error between the high priority patch and the source region patches due to which target patch may fill with unknown information.This is illustrated in the Figure 4. Therefore choosing the suitable similarity measure to fill the target patch is very important.The initial exemplar based image inpainting proposed by Criminisi et al. [25] is enhanced by introducing improved priority computation method and new similarity checking procedure.

Patch Priority calculation
The current example based inpainting method uses the best patches from the source area to fill in the desired area.In this case, we take into account the image's texture and structure data during inpainting.The falling effect is a difficulty with this inpainting technique.This effect prevents accurate inpainting from being performed on the provided images [25].The aforementioned problem is effectively addressed by the use of a superior priority calculation approach that incorporates a regularization factor μ. The priority of the target patch can be derived using equation (4).
where a, b, and c denote the constants associated to the structure information, texture characteristics, and linear structure information in the target region, respectively.The information that is accessible in the source region is used to determine the value of the relevant constant, and that value is determined to be high.The quantity of trustworthy data accessible in close proximity to the pixel  is denoted by the term  � , also known as the confidence term.The following equation was used in the calculation: where  represents the coordinates of the target patch's pixels and �Ψ � � denotes the target patch's size in pixels.The term  � , which reflects the number of isophotes intersecting the border at pixel , is considered.

6
E3S Web of Conferences 472, 02010 (2024) https://doi.org/10.1051/e3sconf/202447202010ICREGCSD 2023 The variable  � represents the geometric properties of the isophote, specifically referred to as the curvature along the direction of the isophote.The introduction of curvature facilitates the propagation of information in the direction of the isophote.The curvature is considered as.
The utilization of isophotes in the computation of the data term resulted in an inadequate definition of the image structures.The eigenvalues derived from the structure tensor matrix effectively capture the pertinent details of the local structural characteristics.The motivation behind this study is to introduce the use of the structure tensor matrix as a means of determining the inpaint location.
The term  � represents the local configuration of the measurement function.The computation is performed utilizing the principles of structural tensor theory.The function is represented as,

Enhanced Exemplar Patch Selection
The exemplar patches, located in the source region, fill the highest priority patch with the maximum priority value.The selection of examplar patch from the source region is accomplished by employing the correlation measure known as the Mean Squared Difference (MSD) between two patches of the source region and highest priority patch.
where, Ψ � is the highest priority patch, Ψ � � exemplar patches in the source region.The MSD is defined by the following equation, The binary mask, denoted as N, is utilized to represent missing content in an image.It assigns a value of 1 to pixels that are missing content and a value of 0 to pixels that are present.Ψ � is employed to extract the pixels that are existing in the target patch, whilst Ψ � � is utilized to extract the matching pixels in the exemplar patch.In essence, the MSD algorithm determines the mean value of the squared differences between comparable pixels located at predetermined positions within two patches.Subsequently, this mean value is employed as a metric to quantify the extent of dissimilarity between the two patches.

Experimental Results and Discussion
The experimentation is taken place on system with configuration intel-core 3 processor with clock speed of 2.7GHz, 12GB RAM using MATLAB software.The efficiency of the proposed method is verified by applying on images taken from Berkeley Segmentation dataset [26].The results are analysed qualitatively by comaring with the state-of-art methods.The comparative results for five different images are shown in figure 5   The enhanced exemplar based image inpainting method is developed by introducing the efficient highest priority computation method and new correlation measurement method.The dropping effect in the highest priority calculation is avoided using this enhanced method and Mean Squared Difference (MSD) is used for exemplar patch selection in the source region.The combination of these two enhanced methods produced the efficient image inpainting results.The obtained inpainting results are more efficient by compring with available methods in the literature.

to 9 .
In each figure,(a)   Input Image; (b) Mask of object to remove; (c) Results from Criminisi et al.[25];(d) Results from Janardhana Rao et al.[13] (e) Results from Yao F et al.[10]; (f) Results from proposed method.The artifacts in the interest of object removed is indicated in box.From the results, visually it clearly understanding that proposed method generated the best quality results compared to the state-of-art techniques in the literature.