Image Processing

YCbCr

Two color spaces are applied in this project, RGB565 and YCbCr. RGB space is applied to complete the image output and YCbCr for image processing. Firstly, RGB signal will be converted to YCbCr color space according to ITU-R BT.601 conversion [1], then, the gray image and skin color can be extracted to complete the following processing steps. For example, the single Y value of this image can represent the gray degree of this image. Moreover, the threshold of YCbCr can be determined to extract the skin color.

Figure 1. Conversion from RGB to YCbCr

After that, lots of noise still existed, hence, the median filter should be applied to eliminate the noise of corresponding image. Each pixel will be sent into median filter entry by entry and value of each entry will be replaced by the median value of neighboring entries. For instance, the neighbors are elements of a 3*3 matrix. Then this matrix will slide entry by entry until the end of this image. Through this method the influence of unexpected noise can be reduced considerably. For instance, an input signal is (2, 3, 80, 6), then, the output of median filter is (y1, y2, y3, y4), y1=med(2,3,80)=3, y2=med(3,80,6)=6, y3=med(80,6,2)=6, y4=med(6,2,3)=3, the final output is (3, 6, 6, 3). Hence, the unexpected noise 80 was eliminated. However, there is a problem relates to the edge property of image, median filter will make a smoother image hence the sharpness of edge will lose.

In this implementation, the median filter has a 3*3 window move from the first entry to the final entry. However, the implementation of calculation of this 3*3 window is a problem because there is no register for the storage of data of each pixel. Hence, the 3*3 shift register has to be formed.

The construction of 3*3 matrix was based on two shift registers and nine 8-bit registers. The output terminal of the first shift register was connected to the input terminal of the second one, as is demonstrated in Figure 2.

Figure 2. Shift registers design

In the shift register, the value flowed to the next byte at the positive edge of the clock signal. Since there were 320 RGB pixel values in a row for a frame, the depth of the shift register was set as 320. In consequence, the value entered the first shift register would be input to the second one after the time period of 320 positive edges of the clock signal.

Nine 8-bit registers were designed to store 9 elements in a 3*3 matrix, as is indicated in Figure 3. The input data could traverse all the registers in the matrix with a specific sequence.

Figure 3. Matrix registers design

With this designing architecture, every input value could move to the next register in the same row at the positive edge of the clock signal. Similarly, the values could flow to the next register in the same column after flowing through the former shift register.

As a result, a series of input values could move through all 9 registers with the following sequence: mat_33 -> mat_32 -> mat_31 -> mat_23 -> mat_22 -> mat_21 -> mat_13 -> mat_12 -> mat_11, which is illustrated in Figure 4.

Figure 4. Data moving sequence

The simulation results were shown in Figure 5. The input data traversed through all the 9 registers with the expected sequence successfully, which verified the design.

Figure 5. Data moved through the third row 

Figure 6.   Data moved through the second row

Figure 7. Data moved through the first row

Firstly, the 3 rows data will be sorted and 3 maximum, 3 median and 3 minimum value will be determined, after that the minimum of these 3 maximum data, the maximum of these 3 minimum and the median of 3 median value will be selected, then the minimum, median and maximum value in the second step will be compared to determine the final median value. Finally, the central entry will be assigned as this final median value. The purpose of comparing all the maximum, minimum and median values again is ensure the final median value is true median value because the maximum value in the first row may less than the minimum value in the third row [2].

Figure 8. Algorithm of sort

In the implementation part, the whole algorithm was divided into 3 parts, however, the content of each module is same, thus, only one module is required. The actual circuit of this algorithm is shown below.

Figure 9. Implementation of sort

Moreover, the erosional data will be delivered into 'dilation' module. The algorithm of this module is assigning the central entry of a 3*3 window with 1 if any entry of this 3*3 window is 1. Hence, the vacancy of an object inside will be filled to make the image more solid and increase the subsequent recognition [3]. 

Figure 10. Implementation of dilation

After that, the image will be delivered into erosion and dilation module to complete image processing. The object for using erosion processing is eliminating the extra noise, especially for the noise which only exists on single pixel. Every pixel in a 3*3 pixels window should be 1 to assign the central entry as 1. Hence, the edge of image will shrink, and the extra noise will be removed [3].

Figure 11. Implementation of erosion

As the figures shown above, the operation of dilation and erosion is inverse, however, these two operations will be neutralized. The combination will firstly making the image more solid, then the erosion operation will eliminate the noise. Finally, the entire combination of 3 modules can be seen in the circuit diagram.

Figure 12. The structure of Video Image Processor module

Reference:

[1] ITU-R. Recommendation  ITU-R  BT.601-7 . Accessed: Feb. 12, 2020. [Online]. Available: https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-REC-BT.601-7-201103-I!!PDF-E.pdf

[2] R. Fisher et al. "Median Filter". 2003. [Online]. Available: https://homepages.inf.ed.ac.uk/rbf/HIPR2/median.htm

[3] MORPOLOGICAL DIGITAL IMAGE PROCESSING USING FPGA, Shodhganga [Online]. Available: https://shodhganga.inflibnet.ac.in/bitstream/10603/205519/7/07_chapter%204.pdf