1
HMM-based Indic Handwritten Word Recognition using Zone
Segmentation
aPartha Pratim Roy*, bAyan Kumar Bhunia, bAyan Das, cPrasenjit Dey, dUmapada Pal aDept. of CSE, Indian Institute of Technology Roorkee, India bDept. of ECE, Institute of Engineering & Management, Kolkata, India cDept. of CSE, Institute of Engineering & Management, Kolkata, India dCVPR Unit, Indian Statistical Institute, Kolkata, India
Abstract
This paper presents a novel approach towards Indic handwritten word recognition using zone-wise information. Because of complex nature due to compound characters, modifiers, overlapping and touching, etc., character segmentation and recognition is a tedious job in Indic scripts (e.g. Devanagari, Bangla, Gurumukhi, and other similar scripts). To avoid character segmentation in such scripts, HMM- based sequence modeling has been used earlier in holistic way. This paper proposes an efficient word recognition framework by segmenting the handwritten word images horizontally into three zones (upper, middle and lower) and recognize the corresponding zones. The main aim of this zone segmentation approach is to reduce the number of distinct component classes compared to the total number of classes in Indic scripts. As a result, use of this zone segmentation approach enhances the recognition performance of the system. The components in middle zone where characters are mostly touching are recognized using HMM. After the recognition of middle zone, HMM based Viterbi forced alignment is applied to mark the left and right boundaries of the characters. Next, the residue components, if any, in upper and lower zones in their respective boundary are combined to achieve the final word level recognition. Water reservoir feature has been integrated in this framework to improve the zone segmentation and character alignment defects while segmentation. A novel sliding window-based feature, called Pyramid Histogram of Oriented Gradient (PHOG) is proposed for middle zone recognition. PHOG features has been compared with other existing features and found robust in Indic script recognition. An exhaustive experiment is performed on two Indic scripts namely, Bangla and Devanagari for the performance evaluation. From the experiment, it has been noted that proposed zone-wise recognition improves accuracy with respect to the traditional way of Indic word recognition.
Key Words: Handwritten Word Recognition, Hidden Markov Model, Indian Script Recognition
Cite This As1
title={HMM-based Indic handwritten word recognition using zone segmentation}, author={Roy, Partha Pratim and Bhunia, Ayan Kumar and Das, Ayan and Dey,
}
1 http://www.sciencedirect.com/science/article/pii/S0031320316300450 (Pattern Recognition, 2016)
1. Introduction
Although, the automatic recognition of printed text has achieved a great success rate, the performance of handwritten word recognition is not high. Handwritten word recognition has long been an active research area because of its vast potential applications. Some of its potential application areas are postal automation, bank cheque processing, automatic data entry, etc. The main hindrance behind the difficulties of making a handwritten recognition system is the huge variation in writing style and complex shapes of characters in words. There are many research works towards handwritten word recognition in Roman , Japanese/Chinese [2, 3] and Arabic scripts . Although many investigations have been made towards the recognition of isolated handwritten characters and digits of Indian scripts , only a few pieces of work [6, 7] exist towards offline handwritten word recognition in Indian scripts.
Devanagari and Bangla are two most popular Indian scripts. Devanagari, script is used to write languages such as Sanskrit, Hindi, Nepali, Marathi, and many others. It is used by approximately 400 million people in northern India and it is the most widely used Indic script. Bangla is the second most popular language in India. Languages like Bangla, Assamese and Manipuri languages are written in Bangla script. About 200 million people of Eastern India and Bangladesh use Bangla script for communication. Also, Devanagari is the third most and Bangla is the fifth most popular language in the world . Examples of Bangla and Devanagari handwritten document images are shown in Fig. 1.
(a) (b) Fig.1: Examples of handwritten text document. (a) Bangla, (b) Devanagari The OCR involving printed Devanagari and Bangla scripts has been addressed in many pieces of research work [9, 10, 11]. Although a number of work has been investigated for isolated handwritten character and digit recognition in Indian script , only a few pieces of work exist towards handwritten word recognition in Indian script [6, 7]. Offline recognition of handwritten word of these scripts needs lot of research.
Most of the existing works in these two scripts are performed on segmenting the characters from words and then recognition. A number of works have been performed for character level segmentation in Devanagari and Bangla . It is reported that due to the presence of noise, touching, etc., the segmentation of characters from a word may often fail. Often characters may generate disjoint character components through preliminary segmentation process. Overlapping and touching characters, which frequently occur in Bangla writing style, create more hindrance in segmenting characters of the words.
3
In the past decades stochastic approaches such as Hidden Markov Models (HMMs) have been widely applied to perform word recognition task [13, 14] because of its effectiveness for modelling unconstrained character-string. This is mostly due to their ability to cope with non-linear distortions and incomplete information. Mainly two approaches namely segmentation-based approach and holistic approach are used for the word recognition purpose. In practice, a HMM can be employed to represent a whole word or, alternatively, sub-word units such as characters which can be concatenated to form general strings. Though HMMs-based techniques have been successfully used in handwriting recognition , only a few papers exist for Indian script recognition. One of the reasons could be the larger number of character classes in Indic scripts due to modifiers and compound characters.
Only a few pieces of work using HMMs are performed in Devanagari and in Bangla handwritten word recognition. Almost all these methods consider holistic approach of recognition as word-wise HMM model creation. In these approaches feature extraction was performed from the entire word and recognition was performed with the help of lexicon-based holistic word recognition. The main drawbacks of these holistic word-based HMMs models are that the recognition process is limited to a set of words only. Also, in this method, for each word a large number of training data is needed.
An unknown word which was not trained by the models will not be recognized using these systems. To overcome these drawbacks, HMMs are trained on sub-word units, such as characters, which can be concatenated to form general strings. Character based HMM models have been successfully used for recognition of arbitrary set of words in English/Latin scripts. One of the advantages is that they allow recognizing unknown words from training data once the character models are trained. HMMs avoid the problem of pre-segmentation of words into characters so that the errors of pre-segmentation can be eliminated. Character alignment based techniques for HMM is also studied to reduce the error . Note that such approach was not applied for offline word recognition of Indic scripts earlier.Though, this character based HMM models are popular in the literature of word recognition, the process may not be directly useful in Indic scripts, especially in Devanagari and Bangla. It is due to the fact that in such scripts, combination of vowels, modifiers and characters lead to a huge number of character classes. Thus, sufficient data for each combination will be necessary for training the respective class models. To reduce such huge number of character classes we propose a zone-wise recognition approach where a word is segmented into 3 zones (upper, middle and lower zone). To have an idea about such character class reduction, let X, Y and Z be the number of character classes that may appear in upper, middle and lower zones, respectively. If we do not use zone-wise recognition then number of character classes will be XY+YZ (assuming all characters in the middle zone may be associated with all characters of upper and lower zones). Whereas if we use zone-wise segmentation, total number of character classes will be X+Y+Z instead of XY+YZ. Thus, there will be a huge reduction of characters when X, Y and Z are large. To have an idea, in Bangla we have about 280 characters (simple and compound together) which may appear in middle zone and 4 modifiers in upper zone and 3 modifiers in lower zones. Thus if we do not use zone segmentation, we will have ideally 280×4+280×3= 1960 classes, whereas after zone segmentation we will have only 287 classes, Thus a reduction of 85.36% can be achieved. Based on this principle, recently we proposed a zone-wise recognition approach and showed some preliminary results. This paper is an extension of the earlier paper including several additional contributions. The main contributions of this extended paper are the following: 1) integration of water reservoir concept for better zone segmentation in a word image, 2) efficient PHOG features developed to improve the performance of HMM based middle zone
4
recognition, 3) the proposed framework has been generalized and tested for Bangla and Devanagari scripts recognition. Overall organization of the rest of the paper is as follows. Section 2 describes some important properties and challenges in Bangla and Devanagari scripts. In Section 3, we describe the pre- processing tasks of Indic word images. The word-recognition framework using zone-wise segmentation results is explained in Section 4. Here details of zone segmentation, feature extraction and recognition approaches are discussed. We demonstrate the performance of the proposed approach in Section 5.
Finally, conclusions are presented in Section 6.
2. Properties Of Bangla And Devanagari Scripts
In Devanagari script, a total 49 basic characters exist, out of these 11 are vowels and 38 are consonants. The alphabet of the modern Bangla script consists of 11 vowels and 39 consonants. The basic characters of Bangla and Devanagari scripts are shown in Fig.2. It can be noted that most of the characters in Bangla and Devanagari scripts have a horizontal line (called Matra/Shirorekha) at the upper part and a baseline. When two or more characters sit side by side to form a word, these horizontal lines generally touch and generate a long line. Characters typically hang from the Matra when written. All Indic scripts run left to right, although some combining glyphs appear to the left of their base character for display. In both Bangla and Devanagari scripts a vowel following a consonant takes a modified shape and placed at the left, right, both left and right, or bottom of the consonant.
These modified shapes are called modified characters. Examples of modified character are shown in Fig.2(i.b) and Fig. 2(ii.b) for Bangla and Devanagari scripts respectively. These modifiers add extra difficulty in the character segmentation procedure of Bangla and Devanagari scripts because of their topological position. A consonant or a vowel following a consonant sometimes takes a compound orthographic shape, which we call as compound character. For details about Bangla and Devanagari scripts, we refer .
(Ii)
Fig.2:Few examples of vowels, modifiers, consonants and conjuncts in (i) Bangla & (ii) Devanagari Script
5
A Bangla or Devanagari word can be partitioned into three zones. The upper-zone (ZU) denotes the portion above the Matra, the middle zone (ZM) covers the portion between Matra and base-line, the lower-zone (ZL) is the portion below base-line. Different zones in a Bangla word image are shown in Fig.3.
Fig.3: Three zones of Bangla script – upper, middle and lower zone separated by Matra and base line 2.1. Challenges in Bangla/Devanagari Word Recognition As discussed above, recognition of Bangla and Devanagari script is not similar like Latin script due to the variation of character-modifiers presence in 3 zones: upper, middle and lower zones. When the consonant character, "ক"(appear only in middle zone)get combined with a vowel, the vowel forms a modifier which can appear either in middle zone (like "ক "), or in middle and upper zone (like " ক") according to the nature of vowel (as shown in Fig.2). Hence, the combinations of consonants and vowel make a large number of possible character combinations. Because of this, traditional HMM-based recognition systems (without zone segmentation)have to consider different character units for each combination separately, as the basic sliding window feature needs to capture the information in all zones for identifying the modifier properly. In Fig.4 (left column), it is graphically shown when a Bangla consonant character „ক‟ is combined with 5 different vowels. In our proposed zone- segmentation based approach (right half of the Fig.4), it is possible to make these complex character shapes to model by few simple character units. For example, we can divide the shape ক into simple
Units ক, , And
.Similar is the case for other consonants of Bangla and Devanagari modifiers. Fig.4: example of character units reduction using zone segmentation
6
Also, while writing Bangla/Devanagari characters, they suffer from distortions depending on the writing style of the person like other scripts. Often the Matra appears missing in handwritten words which also add challenges for developing a generalized recognition system. It is observed that due to the presence of noise, touching characters, etc., the segmentation of characters from a word may fail.
Often characters may generate disjoint character components through preliminary segmentation process, which creates problem in recognition tasks. Proper classification and reunification of these components using segmentation are not easy to process. Overlapping and touching characters, which frequently occur in Bangla/Devanagari writing style, create more hindrance in recognizing characters of the words. Another problem is the “Slant and Skew” nature of handwritten word (see Fig.5). Due to non-uniform skew and slant in word images the recognition of words become more difficult. As mentioned earlier, the “Matra” stays in a horizontal line dividing the upper and mid-section of the word, which often fails to be so. Our recognition framework is designed to take care of these issues.
These are discussed in the following section. Fig.5: (a) Skewed word. Red lines are the “Matra” and “Base line” respectively which are no longer horizontal. (b) Slanted word. Character segmentation lines are not vertical.
3. Preprocessing
To extract the word image from the handwritten document a set of pre-processing tasks are followed. The offline document image is first converted into binary image using global histogram-based Otsu binarization method. The binary document is segmented into individual text lines using a line segmentation algorithm . Here, some seed components of a line are obtained from smoothed text regions of document. The upper and lower boundary information of a text line is obtained from background regions using morphological functions. Next, foreground seed components and boundary information are used to segment the text-lines. Once lines are separated, Run Length Smoothing Algorithm (RLSA) is next applied on each text line to get individual words as a component. A connected component labeling is applied to find the bounding box of the word patches in the line.
Next, using the patch mask, the original word is considered from the binary image. The word images are next processed for skew and slant corrections. These are discussed in following subsection.
3.1. Skew And Slant Correction
In our framework "Water reservoir" concept has been applied for skew correction of non-horizontal words. This idea has been used earlier for various document image analysis techniques such as, script identification, line segmentation, etc. in Indian scripts [6, 10, 21]. In this concept, if water is poured from a side of a component, the cavity regions of the component where water will be stored are considered as reservoirs of the component. Because of touching through head-line in Bangla and Devanagari scripts, two consecutive characters in a word create large cavity regions (space) between them and hence we get large reservoirs from the cavity regions. Water reservoir concept is not new but here we explore its application towards skew detection in Bangla and Devanagari scripts. Details of water reservoir and its different properties can be obtained in .
7
To use the concept of water reservoirs stated above to detect skew angle, we obtain the bottom reservoirs of the word image (by pouring water from the bottom of the word). Then we filter out the reservoirs having lower heights, i.e. less than 3xSw, where Sw is the average stroke width of the handwritten word image. Stroke width (Sw) is calculated as statistical mode of the run lengths of the word‟s foreground. A word image is, at first, scanned row-wise (horizontally) and then column-wise (vertically) to compute foreground pixel‟s run-lengths and their occurrence frequencies. Next, the statistical mode value of these run-lengths provides the estimated stroke width (Sw).
The local minima points (depth-points) from the valid reservoirs having heights more than 3xSw are determined by traversing the contour. Let, B be the set of all such points. Next, a first order degree polynomial (i.e. a straight line) using Linear Regression is computed using points of B. We have noted that the slope (θ) of the calculated line provides a quantitative measurement of the skew nature of the handwritten word image. Thereafter the image is rotated in opposite direction by θ for skew correction.
This process is illustrated in Fig.6(a). The slant angle is next determined and corrected using the vertical projection histogram and Wigner– Ville distribution . Using this projection histogram analysis, we find the height of the peaks in vertical projection analysis at an angle with an interval from -45° to +45° after doing shear transform.
Next, the angle at which clear peaks and troughs are found is considered as slant angle. This is illustrated in Fig.6(b).
(B)
Fig.6:(a) Skew correction using water reservoir. Water reservoirs are computed and checked for consideration according to reservoir height. Next a regression analysis is performed on depth-points of the reservoirs and skew angle is obtained. (b) Slant correction was performed using vertical projection analysis.
4. Proposed Zone-Based Word Recognition Approach
In this section we present our zone segmentation based Indic word recognition framework. We have used the combination of HMM and SVM based classification for handwritten word recognition. After performing the preprocessing tasks, the word image is passed through zone segmentation module.
Unlike, traditional Indic handwritten word recognition approaches, by segmenting the words into 3 zones we reduce the number of basic units for character recognition (discussed in Section 2). After segmenting into zones, recognition of middle zone components is performed using HMM. Upper and lower zone components are recognized using SVM classifier and finally zone wise results are combined to get final results. The proposed zone-based recognition system outperforms traditional character-wise word recognition approach. We also demonstrate that the refinement in zone- segmentation improves the recognition performance. The architecture of the system has been shown in Fig.7 with a Bangla word example. The details of these steps are discussed briefly in following subsections.
4.1. Zone Segmentation
After rectifying the skew and slant defects, the words are segmented into 3 zones: upper, middle and lower zones. For this purpose, the Matra in a word is first detected. Due to complex writing, exact Matra detection is not easy always. Hence we detect possible regions of Matra. These are explained as follows.
4.1.1. Matra Row Detection:In literature of printed word recognition , Matra is usually determined by projection analysis in horizontal direction and considering the row with highest peak. But, due to the free flow nature of handwriting the Matra is rarely a perfect straight line. It is often curvy and broken. To determine the estimated location of Matra, we considered three different row information for locating the approximated row in the word image. Next, the best one among these three is chosen. First row, denoted as R1, is the highest peak determined by projection analysis of the word in horizontal direction. Second row, R2, is the row calculated from depth-points of water reservoirs where the sum of the squares of the distance between this row and each of the depth-points is minimum.
The third row, R3 is computed as follows. Since, the upper zone of Devanagari/Bangla script contain fewer components than that in the middle zone, the portions below the Matra will be more dense than that of above the Matra. Hence, there will be a sharp decline in projection peak in upper half of the word while moving from below the Matra to above the Matra. We mark the row where a sharp decline in projection is observed as the third estimated location (R3) of Matra. Finally, the Matra row has been detected by following rules. These rows are shown in Fig. 8.
Where, Th = H/10, is the threshold and H is the height of the word image; H is calculated by taking mode of height list taken from top and bottom most pixels of each column of the word image. We have noticed that, the location of Matra row has been efficiently detected with this rule in most of the word images.
R1, If | R1- R2| ≤Thor | R1- R3| ≤Th
R2, if | R2- R3| ≤ Th and| R1- R2| >Th and | R1- R3| >Th ...(1)
(C)
Fig.8: An example of Bangla word and corresponding rows for Matra detection. (a) Row R1 is marked along with the projection analysis of the word in horizontal direction, (b) Bottom water reservoirs (shaded in gray) are shown. Depth points are denoted in green peaks. Row R2, obtained by regression analysis of these peaks is marked in red dotted line. (c) Row R3 is marked by black line.
4.1.2. Upper Zone Segmentation: After estimating the Matra row, we create a window of Matra region (WM) of height 4xSw keeping the Matra in middle, where Sw is the stroke width (discussed in Section 3). It is noted that the curvilinear Matra resides in WM in more than 98% of the words from experiment dataset. Next, we extract the skeleton of the word image and find the high curvature points, junction points, and end points of the skeleton image (See Fig. 9). These points are marked as „P‟.
Now, we find the lines between consecutive „P‟s in horizontal direction within WM. If any line-segment emerging from point „P‟ and crosses WM, we consider it as a character-portion and hence discard it. Only those line-segments between „P‟s which are passed within WM are considered. If more than one pixel is found in a single column we consider the upper most pixel. In some words Matra may be broken and discontinuous. There, we join the two nearest Matra pixels using standard Bresenham algorithm. Next the modifiers in the upper zone are marked by checking the upper portions of Matra.
Fig.9: Examples of Matra and Matra region detection. (a) Red line is the Matra row. Yellow lines specify the Matra region. Red squares represent high curvature points, corners and junction points within this frame. Blue line denotes the detected Matra. Green line joins the nearest Matra pixels in case of broken Matra. (For better visibility please see the soft copy of the
Pdf Version Of The Paper)
4.1.3. Lower Zone Segmentation: In our earlier approach , to detect the modifiers in lower zone, we marked the baseline which separated the middle zone from lower zone by observing a sharp decline in the busy zone in lower half of the image. This approach may fail sometimes in situation, when the baseline that separates the middle zone from the lower zone is difficult to locate. If the letters of the
11
word are irregular in size, or there exist many modifiers in lower zones, then we may not find any sharp decline in projection peak between the middle zone and lower zone of the image. To have an idea we show an example in Fig. 10 where due to complex writing style, the lower zone detection becomes difficult using projection analysis.
Fig.10: Examples of cases where baseline detection may fail to segment the lower zone To overcome this problem we include a shape matching based algorithm for modifier extraction in this paper. To segment the lower zone modifiers, we search for modifiers in lower half of the image by shape matching. To do so, we find the touching location of modifier by skeleton analysis and separate them from middle zone. If the residue shape components is matched with any of the lower zone characters with high matching confidence, that part is separated from middle zone. The flowchart of lower zone modifier separation is described in Fig.11. The segmentation is discussed below in details.
Fig.11: Modifier segmentation from lower portion of the image Let M1 be the word image. The skeleton image of M1 is first obtained and the junction and end points in skeleton image are detected. Let L1 be the lower half of the image M1. Using connected component (CC) analysis in L1, the components which are not connected to M1 are detected as modifiers. 8- connection connectivity was used in CC analysis. Some of the lower modifiers can be touching to M1.
To separate these modifiers, the skeleton of the components are traced from lower end points. The intuition behind tracing is that usually, all lower modifiers have one end point in lower half of the
, The Junction Point Is
always a part of a loop. If loop is found, we continue the tracing to detect the next junction point for segmentation. If more than one endpoint is found in a column the lowermost of it is considered in segmentation analysis. After segmentation, given an image portion, we compute the recognition confidence using SVM classifier to obtain the corresponding class label. Details of feature extraction and recognition using SVM are given in Section 4.3.2. The probability score is calibrated using Platt
12
scaling from SVM score [23, 24]. Radial Basis Function (RBF) kernel was chosen in SVM for better performance.If a component is recognized by the SVM with a high confidence (more than 0.6), we consider it as a modifier. If the difference between the top two recognition scores of a component is high, it is also considered as a modifier. In Fig. 12, we show an example of lower zone separation from a Bangla word image.
(a) (b) (c) Fig.12: (a) M1 with horizontal line separating the lower half from upper half. (b) Skeleton of L1 with end points marked in green and junction points marked in red. (c) Lower zone separated out from word.
4.2. Feature Extraction
The middle zone is the primary portion in Devanagari and Bangla word region where characters are often touching with each other. We apply Hidden Markov Model (HMM) based stochastic sequential classifier for recognizing the touching components in this zone.
For HMM-based middle zone recognition, we have developed an efficient feature extraction technique PHOG using multi-resolution HOG features . To measure its effectiveness we have implemented 4 different state-of-the-art approaches and compared their performances. In our previous work we used the LGH (Local Gradient Histogram) feature for feature extraction. Here, we study other features, mainly, profile feature, GABOR feature and G-PHOG feature (combination of Gabor and PHOG) for middle zone recognition. These features are briefly described below.
4.2.1. Phog Feature:
PHOG is the spatial shape descriptor which gives the feature of the image by spatial layout and local shape, comprising of gradient orientation at each pyramid resolution level. To extract the feature from each sliding window, we have divided it into cells at several pyramid level. The grid has 4Nindividual cells at N resolution level (i.e. N=0, 1,2..).Histogram of gradient orientation of each pixel is calculated from these individual cells and is quantized into L bins. Each bin indicates a particular octant in the angular radian space.
The concatenation of all feature vectors at each pyramid resolution level provides the final PHOG descriptor. L-vector at level zero represents the L-bins of the histogram at that level. At any individual level, it hasLx4Ndimensional feature vector where N is the pyramid resolution level (i.e. N=0, 1, 2….).
Dimensional Feature Vector, Where K Is The
limiting pyramid level. In our implementation, we have limited the level (N) to2 and we considered 8 bins (360º/45º) of angular information. So we obtained (1 8) + (4 8) + (16 8) = (8+32+128) = 168 dimensional feature vector for individual sliding window position (See Fig.13).
4.2.2. Lgh Feature:
LGH feature , proposed by Rodriguez and Perronin, was similar to HOG feature for object recognition. A sliding window of fixed width is being shifted from left to right of the word image with an overlapping between two consecutive frames. Next, feature is computed from each sliding window by dividing into 4x4 cells. From each cell, Histogram of Gradient(with 8 bins) is computed and the final feature vector is the concatenation of 16 histograms which gives a 128 dimensional feature vector
13
for each sliding window position. The image is smoothed by Gaussian filter before feature extraction for better gradient information.
4.2.3. Gabor Feature:
The GABOR features has been applied successfully in character and word recognition . Here for our work Gabor filtering in four orientations (0, 45, 90 and 135) is applied and then we used the magnitude as the response for feature extraction. After filtering, the image frame is divided equally into 12 rows. Next, we concatenate the features in each grid to have 48dimensional Gabor features.
4.2.4. G-Phog Feature:
We have also made an experiment with a combination of Gabor and PHOG features called G-PHOG feature. The idea of G-PHOG is motivated from the work of where Gabor feature has been combined to improve the result. From the experiment using G-PHOG feature, it is noted that the efficiency of Gabor feature can be improved by combining it with PHOG descriptor.
4.2.5. Marti-Bunke Feature:
The profile feature proposed by used extensively for Latin script recognition, consists of nine features computed from foreground pixels in each image column. Three global features are used to capture the fraction of foreground pixels, the centre of gravity and the second order moment.
Remaining six local features comprise of the position of the upper and lower profile, the number of foreground to background pixel transitions, the fraction of foreground pixels between the upper and lower profiles and the gradient of the upper and lower profile with respect to the previous column, which provides dynamic information.
Single sliding window position shown with red rectangle
(C)
Fig.13: PHOG feature extraction from a Bangla word image where feature vector at each resolution level is concatenated to give the final PHOG descriptor. Gaussian smoothing applied and feature vector at (a) 0thpyramid resolution level (b) 1st pyramid resolution level and (c) 2nd pyramid resolution level
4.3. Recognition
4.3.1. Middle Zone Recognition using Hidden Markov Model We extracted each of the above feature descriptors using sliding window and apply HMM for word recognition. The feature vector sequence is processed using left-to-right continuous density HMMs . One of the important features of HMM is the capability to model sequential dependencies. An HMM can be defined by initial state probabilities π , state transition matrix A =[aij], i, j=1,2,…,N, where aij denotes the transition probability from state i to state j and output probability bj(OK) modeled with continuous output probability density function . The density function is written as bj(x), where x represents k dimensional feature vector. A separate Gaussian mixture model (GMM) is defined for each state of model. Formally, the output probability density of state j is defined as
( ) ( )
where, is the number of Gaussians assigned to j. and ( )denotes a Gaussian with mean and covariance matrix Σ and is the weight coefficient of the Gaussian component k of state j. For a model λ, if O is an observation sequence O = ( , ,.., ) which is assumed to have been generated by a state sequence Q= (Q1, Q2,.,QT), of length T, we calculate the observations probability or likelihood
( ) ( )
Where is initial probability of state 1.In the training phase, the transcriptions of the middle zone of the word images together with the feature vector sequences are used in order to train the character models. The recognition is performed using the Viterbi algorithm. For the HMM implementation, we used the HTK toolkit .The parameters like, numbers of Gaussian Mixture and state are fixed according to validation data.
4.3.2Support Vector Machine (SVM) and Modifier Recognition The isolated components which were included in upper and lower zones are segmented using connected component (CC) analysis and next they are recognized and labelled as text characters. If the components are broken, an algorithm due to Roy et al. is applied to join the broken contours in the
(For
Devanagari) are separately considered for classification so that the chances of error can be minimized. After resizing the images to 150x150, PHOG feature of vector length 168 is extracted from upper and lower zone modifiers. PHOG feature is considered as it provided better result in the experiment. Next, Support Vector Machine (SVM) classifier has been used to classify these components.
SVM classifier has been chosen here as it has successfully been applied in a wide variety of classification problems. Given a training database of M data: {xm| m=1,..,M}, the linear SVM
(
Where, xj is the set of support vectors and the parameters j and b have been determined by solving a quadratic problem. A linear kernel can be used to classify data which have fewer variations. But changing the kernel function to Radial Basis Function (RBF)was a better choice in our experiment study to classify upper and lower zone modifiers for recognition.LIBSVM toolbox was used for SVM based learning. We used grid search technique to optimize the gamma and multiplier parameter in the library. In grid-search we started with a coarse value of parameters, and then a finer grid around the best parameter values was used.The SVM prediction result is used afterwards to merge with middle zone recognition results to form the entire word.
4.3.3. Combination of Zone-wise Recognition Results In this section, the details of the modifier alignment and combination with the middle zone results are discussed. For estimating the boundaries of the characters in the middle zone of a word, Viterbi Forced Alignment (FA) has been used in the middle-zone of the word. With the embedded training of FA, the optimal boundaries of the characters of the middle-zone are found. After obtaining the character boundaries in the middle zone, the respective boundaries are extended in the upper and lower zones to associate characters present in upper and lower zones with the middle zones characters. This is one hypothesis for characters segmentation and combination of a given word. Similarly, we generate N such hypothesis using N-best Viterbi list obtained from middle zone of the word.
The score to generate a hypothesis is calculated based on the recognition results of middle zone. For a given word image (X), its score is calculated based on a lexicon (W) of the middle zone characters and it is theposterior P(W|X). Using logarithm in Bayes‟ rule we get
( | ) ( | ) ( ) ( )
From these scores N-best hypothesis are chosen. Now among these N-best choices, the best hypothesis is chosen combining upper and lower zone information discussed as follow. After computing the zone-wise recognition results (upper and lower zone modifiers are recognized by SVM and middle zone characters by HMM of a word (X) and recognized character labels are obtained) the labels of upper and lower zones are associated with labels of middle zone. The association of character labels can be considered as a path-search problem to find the best matching word where each character label will be used only once. In our framework, the association is performed as follows. Let,
16
the recognition labels of middle zone characters be CM_1, CM_2 ... CM_N where N is the number of characters obtained in middle zone. Also, let the recognition labels of upper zone characters and lower zone characters be CU_1, CU_2,…,CU_N and CL_1, CL_2,...,CL_N respectively (please note that in most of the cases, number of upper and lower zone characters will be less than middle zone characters). Let these zone-wise character results are obtained and stored in 3 arrays CU[], CL[], and CM[] respectively (See Fig.14(b)). A middle zone character (CM_i) generally be associated to its corresponding upper (CU_i) and lower zone (CL_i) modifies. After association, ideally the whole word will be WT= C1, C2...
CN where Ci= F_i(CM_i, CU_i, CL_i),i = 1, 2, 3...N and F_i is an association function of middle zone character (CM_i) with CU_i and CL_i. But, due to complex handwriting styles, some upper/lower zone modifiers may not appear exactly above and below of their middle zone character (see Fig. 14(a)). To handle such situation, a more flexible association rule is proposed here. In this modified association rule, a middle zone character (CM_i) is associated not only its exactly upper (CU_i) and lower (CL_i) zone modifiers but also CM_i associates with one modifier (CU_i-1), (CU_i) and (CU_i+1) from upper zone and one modifier (CL_i-1), (CL_i) and (CL_i+1) from lower zone. Thus our modified association rule of middle zone character (CM_i) becomes F_i(CM_i,CU_i,CU_i+1,CU_i-1,CL_i,CL_i+1,CL_i-1). Similarly for each middle zone character we find associated upper and lower zone characters and hence associated words are obtained. For each word we may have several associated wordsWT
Number Of Words Formed. Each Associated Word (Wt
j) is matched with the lexicon (L) and best matched associated word is the combined zone-wise result of the word (X). The similarity score in lexicon matching is obtained using Levenshtein distance . By Levenshtein distance matching, we find errors (characters that are not same) between two sequence of characters. The errors are considered as the difference from substitution, insertions, and deletions operation. This string matching algorithm is solved using dynamic programming (DP). Thus, we obtain a distance score for each associated word along with its word selected from lexicon. The scores are next sorted and the lexicon word with minimum score is considered as best result. In Fig.14(b), we illustrate this association process to get combined result. Algorithm 1 details the steps of the combination of zone wise results to find the best matching result.
Ai Adaptive Learning
This project focuses on ai adaptive learning using modern AI and machine learning techniques. The content below is adapted from research literature and practical implementation notes.
We propose a novel high-performance and interpretable canon-
addition, unlike tree learning, DNNs enable gradient descent- ical deep tabular data learning architecture, TabNet. TabNet based end-to-end learning for tabular data which can have a uses sequential attention to choose which features to reason multitude of benefits: (i) efficiently encoding multiple data from at each decision step, enabling interpretability and more types like images along with tabular data; (ii) alleviating the efficient learning as the learning capacity is used for the most need for feature engineering, which is currently a key aspect
salient features. We demonstrate that TabNet outperforms in tree-based tabular data learning methods; (iii) learning other variants on a wide range of non-performance-saturated from streaming data and perhaps most importantly (iv) end- tabular datasets and yields interpretable feature attributions to-end models allow representation learning which enables plus insights into its global behavior. Finally, we demonstrate many valuable application scenarios including data-efficient
self-supervised learning for tabular data, significantly improv- domain adaptation (Goodfellow, Bengio, and Courville 2016), ing performance when unlabeled data is abundant. generative modeling (Radford, Metz, and Chintala 2015) and
Introduction We propose a new canonical DNN architecture for tabular
Deep neural networks (DNNs) have shown notable success data, TabNet. The main contributions are summarized as: efficiently encode the raw data into meaningful representa- enabling flexible integration into end-to-end learning. tions, fuel the rapid progress. One data type that has yet to 2. TabNet uses sequential attention to choose which fea- see such success with a canonical architecture is tabular data. tures to reason from at each decision step, enabling in-
Despite being the most common data type in real-world AI terpretability and better learning as the learning capacity (as it is comprised of any categorical and numerical features), is used for the most salient features (see Fig. 1). This under-explored, with variants of ensemble decision trees for each input, and unlike other instance-wise feature se- Why? First, because DT-based approaches have certain bene- and van der Schaar 2019), TabNet employs a single deep
fits: (i) they are representionally efficient for decision mani- learning architecture for feature selection and reasoning. folds with approximately hyperplane boundaries which are 3. Above design choices lead to two valuable properties: (i) common in tabular data; and (ii) they are highly interpretable TabNet outperforms or is on par with other tabular learn- in their basic form (e.g. by tracking decision nodes) and there ing models on various datasets for classification and re-
are popular post-hoc explainability methods for their ensem- gression problems from different domains; and (ii) TabNet ble form, e.g. (Lundberg, Erion, and Lee 2018) – this is an enables two kinds of interpretability: local interpretability important concern in many real-world applications; (iii) they that visualizes the importance of features and how they are fast to train. Second, because previously-proposed DNN are combined, and global interpretability which quantifies
architectures are not well-suited for tabular data: e.g. stacked the contribution of each feature to the trained model. convolutional layers or multi-layer perceptrons (MLPs) are 4. Finally, for the first time for tabular data, we show signif- vastly overparametrized – the lack of appropriate inductive icant performance improvements by using unsupervised bias often causes them to fail to find optimal solutions for tab- pre-training to predict masked features (see Fig. 2).
ular decision manifolds (Goodfellow, Bengio, and Courville
Why is deep learning worth exploring for tabular data?
One obvious motivation is expected performance improve- Feature selection: Feature selection broadly refers to judi- Copyright © 2021, Association for the Advancement of Artificial ciously picking a subset of features based on their useful-
Professional occupation related Investment related
Feedback from Feedback to
Feature selection Input processing Feature selection Input processing
previous step next step … …
Predicted output (whether the income level >$50k)
selection enables interpretability and better learning as the capacity is used for the most salient features. TabNet employs multiple decision blocks that focus on processing a subset of input features for reasoning. Two decision blocks shown as examples process features that are related to professional occupation and investments, respectively, in order to predict the income level.
Unsupervised pre-training Supervised fine-tuning
Age Cap. gain Education Occupation Gender Relationship Age Cap. gain Education Occupation Gender Relationship 5 2000 ? Exec-managerial F Wife 6 2000 Bachelors Exec-managerial M Husband 1 0 ? Farming-fishing M ? 2 0 High-school Farming-fishing M Unmarried
? 50 Doctorate Prof-specialty M Husband 4 50 Doctorate Prof-specialty M Husband 2 ? ? Handlers-cleaners F Wife 2 0 High-school Handlers-cleaners F Wife 5 3000 Bachelors ? ? Husband 5 3000 Bachelors Exec-managerial M Husband
3 0 Bachelors ? F ? 3 100 Bachelors Prof-specialty F Wife ? 0 High-school Armed-Forces ? Husband 2 0 High-school Armed-Forces M Husband
TabNet decoder Decision making
Age Cap. gain Education Occupation Gender Relationship Income > $50k
3 M False
level can be guessed from the occupation, or the gender can be guessed from the relationship. Unsupervised representation learning by masked self-supervised learning results in an improved encoder model for the supervised learning task.
ward selection and Lasso regularization (Guyon and Elisseeff performance with compact representations. 2003) attribute feature importance based on the entire training Tree-based learning: DTs are commonly-used for tabular data, and are referred as global methods. Instance-wise fea- data learning. Their prominent strength is efficient picking ture selection refers to picking features individually for each of global features with the most statistical information gain
to maximize the mutual information between the selected mance of standard DTs, one common approach is ensembling features and the response variable, and in (Yoon, Jordon, and to reduce variance. Among ensembling methods, random van der Schaar 2019) by using an actor-critic framework to forests (Ho 1998) use random subsets of data with randomly mimic a baseline while optimizing the selection. Unlike these, selected features to grow many trees. XGBoost (Chen and
sity in end-to-end learning – a single model jointly performs recent ensemble DT approaches that dominate most of the feature selection and output mapping, resulting in superior recent data science competitions. Our experimental results
!# + Softmax !" < % !" > % !# > & !# > &
ReLU ReLU &
$" !" − $" % −1 −$" !" + $" % −1 −1 $# !# − $# & % −1 −$# !# + $# & !"
FC FC
W: [$" , - $" , 0, 0] W: [0, 0, $# , - $# ] !" < % b: [-a $" , a $" , -1, -1] b: [-1, -1, -d $# , d $# ] !# < & !" > % !# < & [!" ] [!# ]
M: [1, 0] M: [0, 1]
(right). Relevant features are selected by using multiplicative sparse masks on inputs. The selected features are linearly transformed, and after a bias addition (to represent boundaries) ReLU performs region selection by zeroing the regions. Aggregation of multiple regions is based on addition. As C and C get larger, the decision boundary gets sharper.
for various datasets show that tree-based models can be out- constructs a sequential multi-step architecture, where each performed when the representation capacity is improved with step contributes to a portion of the decision based on the deep learning while retaining their feature selecting property. selected features; (iii) improves the learning capacity via non- Integration of DNNs into DTs: Representing DTs with linear processing of the selected features; and (iv) mimics
DNN building blocks as in (Humbird, Peterson, and McClar- ensembling via higher dimensions and more steps. ren 2018) yields redundancy in representation and ineffi- cient learning. Soft (neural) DTs (Wang, Aggarwal, and Liu Fig. 4 shows the TabNet architecture for encoding tabu- functions, instead of non-differentiable axis-aligned splits. mapping of categorical features with trainable embeddings.
However, losing automatic feature selection often degrades We do not consider any global feature normalization, but performance. In (Yang, Morillo, and Hospedales 2018), a soft merely apply batch normalization (BN). We pass the same D- binning function is proposed to simulate DTs in DNNs, by dimensional features f ∈ <B×D to each decision step, where 2019) proposes a DNN architecture by explicitly leveraging multi-step processing with Nsteps decision steps. The ith
expressive feature combinations, however, learning is based step inputs the processed information from the (i − 1)th step on transferring knowledge from gradient-boosted DT. (Tanno to decide which features to use and outputs the processed ing from primitive blocks while representation learning into sion. The idea of top-down attention in the sequential form edges, routing functions and leaf nodes. TabNet differs from is inspired by its applications in processing visual and text
these as it embeds soft feature selection with controllable data (Hudson and Manning 2018) and reinforcement learn- Self-supervised learning: Unsupervised representation relevant information in high dimensional input. learning improves supervised learning especially in small Feature selection: We employ a learnable mask M[i] ∈ has shown significant advances – driven by the judicious capacity of a decision step is not wasted on irrelevant
choice of the unsupervised learning objective (masked input ones, and thus the model becomes more parameter effi- prediction) and attention-based deep learning. cient. The masking is multiplicative, M[i] · f . We use an attentive transformer (see Fig. 4) to obtain the masks us- TabNet for Tabular Learning ing the processed features from the preceding step, a[i − 1]:
M[i] = sparsemax(P[i − 1] · hi (a[i − 1])). Sparsemax nor-
DTs are successful for learning from real-world tabular malization (Martins and Astudillo 2016) encourages sparsity datasets. With a specific design, conventional DNN building by mapping the Euclidean projection onto the probabilistic blocks can be used to implement DT-like output manifold, simplex, which is observed to be superior in performance and e.g. see Fig. 3). In such a design, individual feature selec- aligned with the goal of sparse feature selection for explain-
tion is key to obtain decision boundaries in hyperplane form, PD which can be generalized to a linear combination of features ability. Note that j=1 M[i]b,j = 1. hi is a trainable func- where coefficients determine the proportion of each feature. tion, shown in Fig. 4 using a FC layer, followed by BN. P[i] TabNet is based on such functionality and it outperforms DTs is the prior scale term, denoting how much a particular feature
Qi while reaping their benefits by careful design which: (i) uses has been used previously: P[i] = j=1 (γ − M[j]), where γ sparse instance-wise feature selection learned from data; (ii) is a relaxation parameter – when γ = 1, a feature is enforced
+ Softmax
Feature Feature …
transformer transformer
x Nsteps Features
+ Softmax
Feature …
transformer transformer Feature Feature Feature Feature transformer
Encoded representation
transformer transformer Attentive transformer … Mask transformer …
Step 2 Decision step dependent
transformer transformer
BN Feature Feature
FC BN transformer transformer
+ 0.5 0.5 0.5 Agg. Agg. Features Features FC FC + +
Reconstructed + … Feature attributes + … features
(a) TabNet encoder architecture (b) TabNet decoder architecture Feature transformer Feature Attentive transformer Shared across decision steps Decision step dependent transformer GLU
Decision step dependent Prior scales
+ 0.5 0.5 0.5
0.5 0.5 0.5
+ Attentive transformer (c) (d)
Prior scales
divides the processed representation to be used by the attentive transformer of the subsequent step as well as for the overall Attentive BN FC
output. For each step, the feature selection mask provides interpretable information about the model’s functionality, and the +
masks can be aggregated to obtain global feature transformer important attribution. (b) TabNet decoder, composed of a feature transformer block at each step. (c) A feature transformer block example – 4-layer network is shown, where 2 are shared across all decision
Prior scales
steps and 2 are decision step-dependent. Each layer is composed of a fully-connected (FC) layer, BN and GLU nonlinearity. (d) +
An attentive transformer block example – a single layer mapping is modulated with a prior scale information which aggregates Sparsemax
how much each feature has been used before the current decision step. sparsemax (Martins and Astudillo 2016) is used for BN FC
normalization of the coefficients, resulting in sparse selection of the salient features. +
to be used only at one decision step and as γ increases, more propose the aggregate.feature importance mask, Magg−b,j = flexibility is provided to use a feature at multiple decision PNsteps ηb [i]Mb,j [i]
PD PNsteps
ηb [i]Mb,j [i].2 i=1 i=1 steps. P is initialized as all ones, 1B×D , without any prior j=1
on the masked features. If some features are unused (as in self- Tabular self-supervised learning: We propose a decoder supervised learning), corresponding P entries are made 0 architecture to reconstruct tabular features from the Tab- to help model’s learning. To further control the sparsity of the Net encoded representations. The decoder is composed of selected features, we propose sparsity regularization in the feature transformers, followed by FC layers at each deci-
form of entropy (Grandvalet and Bengio 2004), Lsparse = sion step. The outputs are summed to obtain the recon-
PNsteps PB PD −Mb,j [i] log(Mb,j [i]+)
i=1 b=1 j=1 Nsteps ·B , where is a structed features. We propose the task of prediction of miss- small number for numerical stability. We add the sparsity reg- ing feature columns from the others. Consider a binary mask ularization to the overall loss, with a coefficient λsparse . Spar- S ∈ {0, 1}B×D . The TabNet encoder inputs (1 − S) · f̂ sity provides a favorable inductive bias for datasets where and the decoder outputs the reconstructed features, S · f̂ . We
most features are redundant. initialize P = (1 − S) in encoder so that the model em- Feature processing: We process the filtered features using phasizes merely on the known features, and the decoder’s last a feature transformer (see Fig. 4) and then split for the FC layer is multiplied with S to output the unknown features. decision step output and information for the subsequent We consider the reconstruction loss in self-supervised phase:
step, [d[i], a[i]] = fi (M[i] · f ), where d[i] ∈ <B×Nd and 2
PB PD (f̂b,j −fb,j )·Sb,j
a[i] ∈ <B×Na . For parameter-efficient and robust learning b=1 j=1
√ PB PB 2
. Normalization b=1 (fb,j −1/B b=1 fb,j ) with high capacity, a feature transformer should comprise layers that are shared across all decision steps (as the same with the population standard deviation of the ground truth features are input across different decision steps), as well as is beneficial, as the features may have different ranges. We decision step-dependent layers. Fig. 4 shows the implementa- sample Sb,j independently from a Bernoulli distribution with
tion as concatenation of two shared layers and two decision parameter ps , at each iteration. step-dependent layers. Each FC layer is followed by BN and eventually connected to a normalized residual √ connection We study TabNet in wide range of problems, that contain with normalization. Normalization with 0.5 helps to sta- regression or classification tasks, particularly with published bilize learning by ensuring that the variance throughout the benchmarks. For all datasets, categorical inputs are mapped
For faster training, we use large batch sizes with BN. Thus, bedding and numerical columns are input without and pre- except the one applied to the input features, we use ghost BN processing.4 We use standard classification (softmax cross (Hoffer, Hubara, and Soudry 2017) form, using a virtual batch entropy) and regression (mean squared error) loss functions size BV and momentum mB . For the input features, we ob- and we train until convergence. Hyperparameters of the Tab-
serve the benefit of low-variance averaging and hence avoid Net models are optimized on a validation set and listed in ghost BN. Finally, inspired by decision-tree like aggregation Appendix. TabNet performance is not very sensitive to most as in Fig. 3, we construct the overall decision embedding hyperparameters as shown with ablation studies in Appendix. as dout = i=1 PNsteps ReLU(d[i]). We apply a linear mapping In Appendix, we also present ablation studies on various de-
Wfinal dout to get the output mapping.1 sign and guidelines on selection of the key hyperparameters. Interpretability: TabNet’s feature selection masks can shed For all experiments we cite, we use the same training, val- light on the selected features at each step. If Mb,j [i] = 0, idation and testing data split with the original work. Adam optimization algorithm (Kingma and Ba 2014) and Glorot then j th feature of the bth sample should have no contribution uniform initialization are used for training of all models.5
to the decision. If fi were a linear function, the coefficient
Mb,j [i] would correspond to the feature importance of fb,j . Instance-wise feature selection
Although each decision step employs non-linear processing, their outputs are combined later in a linear way. We aim Selection of the salient features is crucial for high perfor- to quantify an aggregate feature importance in addition to mance, especially for small datasets. We consider 6 tabular requires a coefficient that can weigh the relative importance samples). The datasets are constructed in such a way that of each step in the decision. We simply propose ηb [i] = only a subset of the features determine the output. For Syn1-
PNd Syn3, salient features are same for all instances (e.g., the
c=1 ReLU(db,c [i]) to denote the aggregate decision con- tribution at ith decision step for the bth sample. Intuitively, if 2
Normalization is used to ensure D
P j=1 Magg−b,j = 1. db,c [i] < 0, then all features at ith decision step should have 3
0 contribution to the overall decision. As its value increases, prove the performance, but interpretation of individual dimensions
it plays a higher role in the overall linear combination. Scal- may become challenging. ing the decision mask at each decision step with ηb [i], we Specially-designed feature engineering, e.g. logarithmic trans- formation of variables highly-skewed distributions, may further
For discrete outputs, we additionally employ softmax during
training (and argmax during inference). An open-source implementation will be released.
Global: using only globally-salient features, Tree Ensembles (Geurts, Ernst, and Wehenkel 2006), Lasso-regularized model, L2X
Syn Syn Syn Syn Syn Syn
No selection .5 ± .0 .7 ± .0 .8 ± .0 .5 ± .0 .6 ± .0 .6 ± .0 Tree .5 ± .1 .8 ± .0 .8 ± .0 .6 ± .0 .7 ± .0 .7 ± .0 Lasso-regularized .4 ± .0 .5 ± .0 .8 ± .0 .5 ± .0 .6 ± .0 .7 ± .0
INVASE .6 ± .0 .8 ± .0 .9 ± .0 .7 ± .0 .7 ± .0 .8 ± .0
Global .6 ± .0 .8 ± .0 .9 ± .0 .7 ± .0 .7 ± .0 .8 ± .0 TabNet .6 ± .0 .8 ± .0 .8 ± .0 .7 ± .0 .7 ± .0 .8 ± .0
output of Syn depends on features X -X ), and global fea- Table 3: Performance for Poker Hand induction dataset. ture selection, as if the salient features were known, would give high performance. For Syn4-Syn6, salient features are Model Test accuracy (%) instance dependent (e.g., for Syn4, the output depends on ei- DT 50.0 ther X -X or X -X depending on the value of X ), which MLP 50.0
makes global feature selection suboptimal. Table 1 shows that Deep neural DT 65.1
TabNet outperforms others (Tree Ensembles (Geurts, Ernst, XGBoost 71.1
and Wehenkel 2006), LASSO regularization, L2X (Chen LightGBM 70.0 van der Schaar 2019). For Syn1-Syn3, TabNet performance TabNet 99.2 is close to global feature selection - it can figure out what Rule-based 100.0 features are globally important. For Syn4-Syn6, eliminating instance-wise redundant features, TabNet improves global feature selection. All other methods utilize a predictive model Poker Hand (Dua and Graff 2017): The task is classifica-
with 43k parameters, and the total number of parameters is tion of the poker hand from the raw suit and rank attributes of 101k for INVASE due to the two other models in the actor- the cards. The input-output relationship is deterministic and critic framework. TabNet is a single architecture, and its size hand-crafted rules can get 100% accuracy. Yet, conventional is 26k for Syn1-Syn and 31k for Syn4-Syn6. The compact DNNs, DTs, and even their hybrid variant of deep neural DTs
representation is one of TabNet’s valuable properties. (Yang, Morillo, and Hospedales 2018) severely suffer from the imbalanced data and cannot learn the required sorting and Performance on real-world datasets ranking operations (Yang, Morillo, and Hospedales 2018).
Tuned XGBoost, CatBoost, and LightGBM show very slight
as it can perform highly-nonlinear processing with its depth, Model Test accuracy (%) without overfitting thanks to instance-wise feature selection.
CatBoost 85.1 Table 4: Performance on Sarcos dataset. Three TabNet mod-
AutoML Tables 94.9 els of different sizes are considered.
Forest Cover Type (Dua and Graff 2017): The task is clas- MLP 2.1 0.14M
sification of forest cover type from cartographic variables. Adaptive neural tree 1.2 0.60M approaches that are known to achieve solid performance (AutoML 2019), an automated search framework based on TabNet-M 0.2 0.59M ensemble of models including DNN, gradient boosted DT, TabNet-L 0.1 1.75M with very thorough hyperparameter search. A single TabNet without fine-grained hyperparameter search outperforms it. Sarcos (Vijayakumar and Schaal 2000): The task is re-
gressing inverse dynamics of an anthropomorphic robot arm.
very small model is possible with a random forest. In the very and TabNet merely focuses on the relevant ones. For Syn4, small model size regime, TabNet’s performance is on par the output depends on either X -X or X -X depending parameters. When the model size is not constrained, TabNet feature selection – it allocates a mask to focus on the indi- achieves almost an order of magnitude lower test MSE. cator X , and assigns almost all-zero weights to irrelevant
features (the ones other than two feature groups). models are denoted with -S and -M. Real-world datasets: We first consider the simple task of mushroom edibility prediction (Dua and Graff 2017). Tab- Model Test acc. (%) Model size Net achieves 100% test accuracy on this dataset. It is indeed Sparse evolutionary MLP 78.4 81K known (Dua and Graff 2017) that “Odor” is the most discrim-
What is this project about?
This project covers practical implementation and research aspects of the topic using AI/ML techniques.