Ansys Plate Fixation Fea

Is Of Growing Importance For Enabling Intelligent

transportation and ensuring the security and safety

Lenge In A Practical Environment. The License

plates can have extremely diverse sizes, fonts and

Even Lighting, Occlusion, And Blurring. In Appli-

cations such as surveillance, it often requires fast

Cense Plate Recognition, In This Work, We Propose

a set of techniques: 1) We introduce a contour re-

Tures Into The Cnn Network, And Exploit Transfer

learning to obtain the initial parameters for more

Verification Procedure To Determine The Correct

plate at low cost, a statistical filtering in the plate

Didates, And The Accurate Cr Results After The Cr

process to perform further plate verification with-

Experimental Results Demonstrate That Our System

can accurately recognize license plate in real-time. Additionally, it works robustly under various lev- els of illumination and noise, and in the presence

System Is Not Only Among The Most Accurate Ones

but is also the fastest, and can be easily applied to other scenarios.

1. Introduction

License plate detection and recognition (LPDR) of vehicles has been playing an increasingly important role in building intelligent societies and cities. LPDR can be exploited to enforce the security of communities, enable road safety, and facilitate the collection of payment for parking or road toll (Hsu et al., 2013). Although there are many recent efforts on LPDR (Zheng et al., 2013; Yoon et al., 2012; Zhou et al., 2012; Anagnostopoulos et al., 2006), existing systems gen- erally cannot recognize the plate at high accuracy while also completing LPDR in real-time. LPDR performance suffers when there is no advanced hardware to capture high-quality images or the recognition is needed for a plate attached to a fast moving car (Hsu et al., 2013).

The difficulty lies in the extreme diversity of character pat- terns. Characters on the plates can be in different sizes, fonts, and colors depending on their states and nations, distorted by the viewpoint of camera, and captured with low-quality im- ages due to lighting, shadows, occlusion, or blurring. LPDR is made even harder if there is a requirement for real-time processing. The highly complicated backgrounds introduce additional challenges, and often lead to false alarms in plate detection. Some example backgrounds include the general texts on shop boards, text-like patterns on the car windows, as well as guardrails and bricks on the road whose textures are similar to license plates.

A complete LPDR process is typically composed of two phases: license plate detection (LPD) and license plate recognition (LPR). LPD aims to identify and localize a license plate, and generate a bounding box around the plate.

Usually, a verification process is also needed to guaran- tee that the plate is correctly detected. In LPR, plates are firstly segmented into several characters, and then each is recognized by a recognizer.

The aim of this work is to achieve LPDR at high accuracy in real-time, and it is critical to have LPD and LPR both efficient. To reduce the running-time in the LPD procedure, we apply the most efficient edge-based algorithm. Despite its quick processing speed, this method is sensitive to noise, and cannot deal with broken edges and remove irrelevant

Recognizing License Plates In Real-Time

edges properly. The segmentation of characters is difficult if there exist faked picture borders. Being able to learn mid and high level features from the training data, convolutional neural networks (CNNs) have shown good recognition per- formances in many vision tasks, such as image classification and object detection (Krizhevsky et al., 2012). Although CNN appears to be promising in recognizing general char- acters, it is often difficult to obtain enough data to well train CNN for LPDR, given the diversity of plate types, the varying quality of images captured during car mobility, and the complex backgrounds around plates. This will cause a CNN to overfit easily. Finally, existing LPDR systems often take separate and independent algorithms for the candidate verification (Muhammad & Altun, 2016; Yang et al., 2015), which introduces additional computation cost.

To address these issues, we implement a complete LPDR system to enable real-time and accurate license plate detec- tion and recognition, with the structure shown in Fig. 1. The

Contributions Of Our Work Are As Follows:

1. We propose a contour reconstruction scheme to quickly localize the plate in the presence of incomplete or dis- torted plate contours in a practical environment.

2. We refine the license plate candidates with a simple zero-one-alternation method to effectively remove the irrelevant edges and noise.

3. We introduce a few strategies in our design to over- come the overfitting problem for more accurate char- acter recognition, including data augmentation, incor- poration of SIFT features to the network, and transfer learning to determine the initial training parameters for the network.

4. We propose a two-phase verification method to deter- mine the correct plate: 1) We first apply a statistical filter in the LPD stage to effectively remove the wrong plates from the candidate set to reduce the processing

Overhead For Further Processing In A Lpdr System;

and 2) We take advantage of our accurate character recognizer to help verify the candidates after the LPR process. Without need of extra technique and time to verify the plate, the second phase is equivalent to short- ening the system pipeline, which further improves the efficiency of our LPDR system.

The paper is structured as follows: we introduce the related work in Section 2. We introduce the LPD and LPR problems, and our methods and CNN model to address the issues in Section 3 and Section 4. The experimental results are presented and analyzed in Section 5. Finally, we conclude the paper and introduce our future work in Section 6.

Figure 1. The LPDR system structure.

2. Related Work

We briefly introduce the previous work which targets for different phases of the LPDR system.

2.1. License Plate Detection

For the license plate detection (LPD), previous studies (Anagnostopoulos et al., 2008; Du et al., 2013) generally try to capture certain morphological, color or textural features of a license plate. They are either computationally expen- sive and thus not suitable for real-time systems, or very easy to be affected by the color change in plates. Hough transform methods (Le & Li, 2006) assume that license plates are defined by lines around them, and require a large memory space and considerable amount of computing time.

Histogram-based approaches (Nejati et al., 2015) do not work properly on the images with big noise or license plates tilted. Learning methods with sliding window (Anagnos- topoulos et al., 2006; Zheng et al., 2013) suffer from high computational cost with their applying the classifier to a sequence of rectangles within an image. In addition, not all objects are box-shaped, and representations may be polluted by features not belonging to the object. Edge-based ap- proaches are the simplest and fastest (Al-Ghaili et al., 2013; Jiao et al., 2009), and are employed in our work. (Laroca et al., 2018; Wang et al., 2022; Usama et al., 2022) pro- posed to use Depthwise Separable Convolution Networks or YOLO Detector for the License Plate Detection task.

However, existing methods are sensitive to unwanted edges often appearing close to the license plates, which may lead to a wrong detection. We propose a simple method which exploits contour reconstruction together with statistical fil- tering to quickly and accurately detect the license plate.

Different from the use of global search such as with sliding windows, our design follows the nature of human visual detection, where an attention is driven to certain locations by low-level features of images, such as contours, edges, and texts, rather than uniformly to all locations.

2.2. License Plate Recognition

The License plate recognition (LPR) step consists of three parts: preprocessing, segmentation, and character recogni-

Recognizing License Plates In Real-Time

tion. In preprocessing, skew correction and image refinement are often applied to deskew the image and remove the unnec- essary borders and noises. Then a projection-based method is applied for segmentation (Al-Ghaili et al., 2013), which often does not work well when there are redundant borders and other edges around the license plate, especially when the faked borders appear at the top or bottom part of the plate. Based on the character features of license plates, we propose a zero-one-alternation method to correctly remove the unwanted borders for more accurate segmentation.

Character Recognition (CR) in a general context has been widely studied. The template matching method (Zheng et al., 2013) is simple and straightforward, but is vulnerable to font, rotation, noise, and thickness changes. SVMs (Dong & Feng, 2014) and shallow BP neural networks (Laxmi & Rohil, 2014) are also popular, but they are not good enough to get the most important information from the characters.

CNN-based methods have been proven to be very efficient in image recognition and classification with their ability of learning richer and higher level representations of features (Krizhevsky et al., 2012; Yang et al., 2019; 2020a; Yang et al., 2014; Feng et al., 2015; Zhang et al., 2020; Yang et al., 2021a; Yang et al., 2020b; Yang et al., 2021b; Liu et al., 2020; Liu et al., 2020). They are invariant under small shifts, distortions, and noise. Despite the potentials, CNNs need a huge amount of training data to avoid “overfitting”, while it is often very hard to get a big training set in real- world applications. With the large variety of license plates and their differences in varying environment conditions, get- ting enough data for training becomes even harder. This will significantly compromise the CR performance. To im- prove the CR accuracy, we propose various schemes for data augmentation, aggregate SIFT features, and exploit transfer learning.

3. License Plate Detection

License plate detection (LPD) is the first critical step of the LPDR system. Our LPD consists of two major procedures: candidate plate detection and candidate plate verification.

Candidate plate is defined as the area that potentially con- tains a license plate. To more effectively detect the candidate image region of the plate, we propose a set of schemes based on edge detection, so that the scheme can better work under different backgrounds, motion blur, light conditions, and tilt angles.

3.1. Candidate Plate Detection

With the edge detection, some of the edges may form closed contours, and one may be around the license plate. A li- cense plate is generally bounded by a rectangle area which

(C)

Figure 2. Candidates of different scenarios and the correspond- ing reconstruction results: (a) a normal candidate. (b) A tilted

Candidate. (C) A Tilted Broken Candidate

contains some texts. However, in a practical environment, the contour of the license plate may not be a rectangle. It may be broken or tilted (skewed). In order to improve the detection accuracy, we first propose to reconstruct the edges to form complete rectangular candidate contours that possi- bly contain the plate. To speed up the detection process, we further propose a set of schemes to reduce the candidate set.

As an intuitive way of finding the location of a license plate, one can first get rectangular contours after the edge detec- tion, and then select the one with the right width, height, and convex area. This method suffers when the contours are broken or greatly tilted. It also cannot be applied to detect the license plates with a wide range of height, width, and area.

To better deal with the plate candidates with broken or highly tilted contours, we propose to reconstruct a com- plete contour around the plate based on the extreme points found on the edge map. As shown in Fig. 2, if we have the top-most point A, the right-most point B, the bottom- most point C, and the left-most point D, we can reconstruct the candidate contour by taking the rectangle A′B′C′D′, where xA′ = xD, yA′ = yA, xB′ = xB, yB′ = yA, xC′ = xB, yC′ = yC, xD′ = xD, yD′ = yC.

With this procedure, no matter the contour is broken, tilted, or concave, we can always reconstruct a proper rectangle around it. For example in Fig.2(c), a broken contour with only two borders is still able to be reconstructed. Although simple, our performance studies indicate that this process helps to significantly improve the accuracy of LPD.

3.2. Coarse Candidate Plate Verification

After the detection process, there may exist multiple can- didate plates. To reduce the later processing overhead, we will filter out the wrong candidates and keep only the ones most likely to be correct. We propose a Statistical Filtering method in this section.

Before presenting our scheme, we first introduce a vectoriza- tion process to translate a plate image containing alphanu- meric characters to a “pixel vector” by summing up the “pixel matrix” along its columns and then normalizing it by

Recognizing License Plates In Real-Time

Figure 3. Candidates and their normalized pixel vectors.

255

. We observe that the pixel vector exhibits certain statistical regularities distinguishable from the background. In the plots of the plate candidates and their 1-D pixel vectors on Fig. 3, we can see that a true candidate has some distinctive statistical regularities different from the false ones. For example, a true license plate generally has a sequence of “peaks” and “valleys”, and the number of peaks is larger than most of the false ones.

Definition 1 A value is considered to be a “peak” if it is a local maximum of the pixel vector and the other minima points on its left and right are smaller than it by a threshold.

A similar definition can be applied to “valley”. In the Fig. 3, suppose the threshold is 3, then there are 8 peaks on the correct plate, while there are only 2 peaks on the wrong plate. A candidate with more than 6 peaks (the number of characters in the plate) will be considered as a potential plate and passed to the subsequent procedures. The others will be eliminated.

4. License Plate Recognition

License plate recognition (LPR) consists of three steps: pre- processing, segmentation, and character recognition (CR). Pre-processing procedures such as skew correction and im- age binarization are commonly taken first to prepare an image for the subsequent steps. A refining procedure is then applied to remove the unnecessary parts (e.g., the bor- dering regions) from the binary image, so it will ideally look like that in Fig.4(b). Finally, with the binary image in Fig.4(b), a pixel vector can be built, as shown by the plot in Fig.4(e). The segmentation is performed along the red dotted lines passing through the valleys of the 1-D vector.

This segmentation method is called vertical projection. The segmented letters are shown in Fig.4(f). With the existence

(F)

Figure 4. Plate preprocessing and segmentation illustration: (a) The binary plate. (b) The refined binary image after ZOA process- ing. (c) Normalized pixel vector for (a). (d) Number of zero-one- alternations along rows. (e) Normalized pixel vector for (b). (f) The segmented characters.

of noisy borders around the plate, the segmentation step is often difficult. Although the left and right borders can be removed through the literature method (Yoon et al., 2012), existing methods often fail to remove the top and bottom borders in the presence of distracting image contents. This may lead the segmentation error. We propose a simple zero- one-alternation (ZOA) method which can effectively and quickly remove the top and bottom borders. After the char- acters are segmented from the license plate, a robust CR method that can work in different situations is followed to better recognize the segmented characters.

4.1. Refinement With Zero-One-Alternation

A binary image after the pre-processing has two values, 1 (255) or 0. To more effectively remove the noisy top and bottom borders outside the characters of the license plate, we propose a novel scheme called zero-one-alternation (ZOA).

Definition 2 Along each row of the binary image matrix, the number of changes from 0 to 1 or from 1 to 0 is the number of zero-one-alternations (ZOAs).

By scanning row-by-row from the top to the bottom of the binary image, if we can observe similar number of ZOAs (more than α) from row i until row i + n, with n ≥β, it is very likely that the characters are contained within a region of n rows. In Fig.4(d), this region is from point 1 to point 2.

α and β are the only parameters need to be predefined. α

Recognizing License Plates In Real-Time

and β can be set by the smallest number of alphanumeric characters and the lowest height of the characters in pixels in all the license plates. In this work, we set α = 6 and β = 8. Note that these two are heuristic parameters, but should work well in most cases.

After removing the top and bottom borders using ZOA, we get Fig.4(b). Without this step, the plate might be mistakenly recognized as “7E67Q9” instead of “7F6709”. To remove the left and right borders, we further apply a commonly used method in (Yoon et al., 2012). The segmentation can be efficiently performed over the properly trimmed image through the vertical projection introduced earlier.

4.2. Character Recognition

The last key phase of a LPDR system is to recognize the segmented characters of the license plate. It is necessary and important to propose a robust CR method that can work in different situations.

Due to the large variety of license plates and their images differ in varying environment conditions, there is no specific dataset for plate character recognition. Thus, we create a dataset by ourselves, as introduced in Section 5.1. However, the training data available are far from enough. This will significantly compromise the CR performance. In our cre- ated dataset, we have 36 classes and 3,000 examples in total.

So on average less than 100 training data are available for each class, where a typical classification problem requires more than 1,000 training data for each class. Thus, the character recognition for license plates is challenging. With very limited training data, the learning faces the problems of over-fitting, where the trained model does not apply well to new data.

To enable higher quality CR for LPDR system, we will exploit the use of data augmentation, a newly designed SIFT-CNN model, and the transfer learning.

4.2.1. Data Augmentation

In order to make full use of the limited training samples, we consider two methods for data augmentation. In the first method, we augment data via a number of random affine transforms to avoid the duplication of data in the training set. We gradually increase the amount of augmentation till we have a low testing error. This helps to alleviate the overfitting problem and make our model more easily applied to new scenarios.

A big challenge of performing accurate CR for license plate is that a plate image is often subject to different illumination and noise. In the second method, we vary the illumination of our training data and inject Gaussian noise into our model so that it performs more robustly under different image quality.

Our augmentation parameters are summarized in table 1.

Cnn

CNN requires massive training data to work well in differ- ent scenarios. As an another scheme, scale-invariant feature transform (SIFT) method (Lowe, 1999) can extract image descriptors for recognition without requiring a large amount of data. SIFT searches an image at multiple scales and positions to look for the regions that have the maxima or minima with high contrast, called keypoints. The image gradients can be tracked with a histogram and represented as a descriptor to characterize the appearance of a keypoint.

To describe a keypoint, the region around the keypoint is di- vided into 4×4 subregions, within each gaussian derivatives are computed in 8 orientation planes. So a 128-dimension descriptor vector is formed for each keypoint, and this pro- cedure is repeated for all keypoints to obtain a set of vectors for the image. SIFT descriptors are invariant to translations, rotations, and scaling of the image, and robust to moderate transformations and illumination variations. Despite these benefits, SIFT filters are fixed and cannot vary for a different source of data, which compromises its performance. To take advantage of both schemes while avoiding their limitations, we propose a new SIFT-CNN model which integrates the vector extracted by SIFT with the feature vector of CNN to increase the accuracy of character recognition.

Our method is inspired by a bag-of-words (BoW) represen- tation where image features are treated as words. In BoW, detection and description of image features are first applied, followed by assigning feature descriptors to a set of prede- termined clusters (like vocabulary). Finally, a vector called a BoW is used to track the number of occurrences of words in the vocabulary.

Fig. 5 shows our SIFT-CNN model, where a SIFT branch is added to extract the image feature vector as follows: 1. Using SIFT method to extract the keypoint descriptors of the image.

2. Assigning each keypoint descriptor to a set of clusters whose centers are determined using K-means based on the descriptors of training data.

3. Constructing the feature vector v for each image to track the number of keypoints assigned to each cluster,

Recognizing License Plates In Real-Time

Figure 5. Network architectures for CR problem. Left: traditional CNN-8 model as a reference. Right: CNN-8 incorporated with SIFT (SIFT-CNN) model.

V

|v|l2 . 4. Concatenating the feature vector v with the feature vector extracted in the fully-connected layer of CNN to form a final feature vector.

We set the number of clusters to 256, so that the length of the feature vector is the same as that of the CNN feature vector. Then the concatenated feature vector has the size 512, and will be used to classify the image.

4.2.3. Efficient Learning Across Applications

Many machine learning methods work well only under a common assumption, the training and testing data are taken from the same feature space and follow the same distribution (Pan & Yang, 2010). When the distribution changes, most statistical models need to be rebuilt using newly collected training data. It would be helpful to exploit an emerging technique called the transfer learning, which extracts the knowledge from one or more source tasks and applies the knowledge to a target task. For a small training dataset, with- out enough knowledge on the data, randomly initializing a neural network can make the training result worse since it is easily to get stuck in local minima. However, if a neural network can start from an already trained feature extractor, it can “borrow” some knowledge from a task already learnt to the new task. Despite the potential, it is crucial and difficult to determine the right knowledge sources and the amount of knowledge to use for a new learning task. If too few are transferred, we may not get enough knowledge from source tasks, while transferring too many parameters will not only lead to redundancy but may also compromise the training performance if many parameters are irrelevant.

To further improve the quality of character recognition, we apply transferring learning to the training of our CNN model. Our preliminary studies indicate that features learnt from the low-level and mid-level of a neural network are more effec- tive for transfer learning. We “borrow” the parameters of a SIFT-CNN model, trained on the dataset ICDAR (Karatzas et al., 2015). The dataset contains more than 62 classes, which include 26 upper-case letters, 26 lower-case letters, 10 digits classes and other classes. Our studies indicate that this dataset is excellent in extracting low and mid level features of the document letters. Initially, we attempted to apply the model trained with this data set directly to recog- nizing the characters on the license plates. However, the CR accuracy is only 84%, since the license plate characters are quite different from the document letters in fonts, types and classes. Instead, as shown in Fig. 6, we train a new model based on license plate data. We “borrow” the convolutional layers 1 −4 of the trained SIFT-CNN model, and rebuild two convolutional layers CA−CB and two fully-connected layers FCA −FCB with the number of neurons being 512 and 36 respectively.

4.2.4. Hybrid Cnn Model

For recognizing characters on license plates, we apply Hy- brid CNN which exploits a SIFT-CNN model with transfer learning. We train our model using stochastic gradient de- scent (SGD) algorithm with an annealed learning rate. We test our model with different number of transferred layers and fine-tuned layers, and we find four transferred layers and four fine-tuned layers can achieve the best result. Thus, we fix the 4 transferred layers and only fine-tune the 4 rebuilt layers using the target training data created for this work.

The training process stops when there is no improvement in performance for 5 epochs. In the experiment section, we will show that the transfer learning together with fine-tuning can improve the recognition performance, especially when the training data are very limited.

If we fine-tune all layers, it will not only take a very long time, but may also break the good feature filters already built by the original model. We only fine-tune the last two convolutional layers and the two fully-connected layers. The features extracted from ICDAR dataset by the low-level and mid-level convolutional layers are very general, and can be easily applied to the classification of our images. The higher convolutional layers trained with the ICDAR dataset, however, do not fit well in our system and cannot provide satisfying CR results.

4.3. Fine Candidate Verification With Voting

After the statistical filtering in the LPD phase, we might still have more than one candidate plate, usually two or three. To

Related Journal Articles & DOI Links

Selected peer-reviewed publications relevant to 12 Lead ECG Acquisition. Click the DOI to access the full paper (may require institutional access).

Why Choose Us?

Bangalore guidance for robotics, Spectre and autonomous systems projects.