A Fever Screening Solution Using Optical and Thermal Cameras

The world is facing an unprecedented situation with COVID-19, and many countries have been forced to go into a state of lockdown to stop the extent of virus spread. As governments and organizations reopen, they require systems which can effectively mitigate risks to their citizens, employees and customers.

The most common symptom expressed by people affected with COVID-19 is a fever. Taking contact temperature provides accuracy in detection but is impractical for high-traffic situations such as people entering an office building. Thermal camera-based fever screening solutions are an attractive, contactless proposition in theory, but in practice they can suffer a few challenges:

Infrared (IR) thermometry readings can vary because of ambient temperature, distance of the subject and location on the face where the readings are taken.
The temperature readings can also depend on bio-physical phenomena such as age, makeup on skin, sweat, etc.
Absence of robust algorithms which work on thermal infrared images to locate specific region(s) of human face.
Throughput of such a detection system, i.e., how long it could take to read temperature.

To overcome some of these problems, data scientists and engineers from WWT's AI R&D team started to develop a contactless fever detection system that uses a thermal camera and could be accurate, practical and cost effective.

Different approaches to feature extraction

To get the most accurate readings, our system relies on sampling the temperature of the eye canthus. The eye canthus is the corner of the eye, closest to the nose bridge, and is considered the best facial location to measure core body temperature [1, 2, 3].

The team used two possible approaches to identify the eye canthus, and a thermal camera to measure temperature:

Thermal-Only: Identify the eye canthus directly from an IR image and measure the temperature.
Optical + Thermal: Detect facial features, including the eye canthus, using an optical image and transport the coordinates to the corresponding IR image and measure the temperature. This is attractive due to already available robust open-source facial feature algorithms in the market.

The thermal-only approach required us to collect more data in order to re-train or fully train a robust deep learning model on IR images, so we chose the latter approach to create a proof of concept (POC) in the shortest time possible, with a view to shifting to the former, more cost-effective, solution once we collect more data.

Camera hardware used

A thermal camera with a 14° field of vision (FOV) lens and 464 x 348 detection array was used as a thermal IR camera. The narrow FOV is necessary to capture a sufficiently detailed IR image of the subject at least one-meter distance.
Intel RealSense D415 which consists of the RGB sensor with 1920 x 1080 resolution and uses active IR stereo technology to provide both optical and depth images.

Solution description

Dataset collection

All camera systems were initially integrated with a data collection application to produce datasets for analysis. Due to requirements of the thermal camera vendor's SDK, this application was written in C# on the .NET platform. These datasets included a combination of IR, optical and depth data. Figure 1 shows examples of the data capture process.

*Figure 1: Optical - IR - Depth sample images captured*

Eye canthus point detection

IR images tend to blur or outright remove features we see in visible light. Hence, optical images were taken to detect the eye canthus point using an off-the-shelf facial feature detection library.

The team experimented with the open-source "face-detection" library in Python which uses a pre-trained convolutional neural network (CNN), but observed a high false negative rate due to people wearing masks. The team switched to a pre-trained "face-landmark-detection" from Intel's OpenVINO model zoo. This model was more robust when dealing with occluded faces, and we were able to use pose estimation to collect a temperature only when the subject is looking toward the camera.

Coordinate transformation

After sorting face landmarks, the next challenge in the approach was to map optical image coordinates to IR coordinates. This transfers eye canthus points from the optical image to the IR image to detect temperature. The two cameras are physically separate and have significantly different optical parameters. Figure 3 below shows the full camera setup on a single tripod mounted together in matching orientations.

*Figure 2: Eye canthus transformation process*

A flat calibration object was 3D printed and heated to appear in both IR and optical images. After some initial attempts by the team at homography transformations, intrinsic and extrinsic matrices were manually constructed for the thermal camera.

Combined with the published intrinsic matrix for the Intel RealSense D415, points can be directly transformed from the RGBD camera coordinate space into the thermal camera coordinate space. The eye canthus transformation process along with each camera matrix used is shown in Figure 2 above.

Temperature measurement

Common thermography guidelines recommend capturing the largest possible sample of pixels from the region of interest to collect a reliable temperature sample [4]. Hence the area of 10 pixels was taken out of the left and right eye canthus regions and max temperature from the two regions was taken as subject's temperature for a particular frame. The final temperature value was calculated by averaging the max temperature value over ten consecutive frames.

Since a black body calibrator [5] (thermal reference source) was not being used, average differences between detected and manually measured temperatures were used to determine the bias between sensor readings and real temperatures. All the temperature readings are then adjusted using this bias.

Proof of concept

Subject privacy

Due to the Health Insurance Portability and Accountability Act (HIPAA) regulation, we make sure subjects are anonymized and identified only by an automatically incrementing number. This identifier increments after a set period has elapsed without a subject being detected in the allowed distance range. Also, the images collected for testing purposes only include the region of the eye canthus captured instead of the complete IR face to eliminate any way in which a subject's identity can be compromised (see Figure 5).

Application description

A Windows application developed to support the overall system is shown below in Figure 4. Live frames captured by both the RGBD (Intel D415 module) and thermal camera are continuously retrieved and displayed to the operator in the left two panes. The image in the right pane updates as new temperature readings are determined. Ten readings are collected, averaged and added with a bias to make a final determination, which is displayed in the "Results" panel of Figure 4.

Subjects are screened one at a time. The system does not collect readings until the subject is detected to be within a 0.5 meter range centered at 1.25 meters, oriented towards the thermal camera and not wearing eyeglasses. Once in position, data collection is completed within approximately two seconds.

The operator is prompted to collect a manual temperature reading from subjects that appear to have an elevated temperature. This prompt is also shown periodically for other subjects. The bias calculated using detected and manually measured temperature is updated continuously and applied to all temperatures shown to the operator. An example of this bias can be seen in the bottom left corner of Figure 4 below.

All measurement data and eye canthus portions of the thermal images are logged to local hard disk storage. When Internet connectivity is available, this data is forwarded to a relational database hosted on AWS and removed from local storage.

Initial proof of concept graphics — *Figure 4: Initial proof of concept*

Initial results

The team targeted WWT's North America Integration Center (NAIC) as an initial host site. The NAIC has a vast and diverse employee base, allowing ample opportunity to collect temperature measurement data. This data set can then be used to calibrate and improve system accuracy for future installations.

Below are some of the sample IR images captured after the solution was deployed. All of these are IR images of resolution 464 x 348 showing the area where temperature is measured. Blacked out parts in the images were not captured in the deployment. This was done so that subject's identity was not compromised, and we strongly comply with the HIPAA regulation.

*Figure 5: Sample images captured from NAIC*

Learnings & key takeaways

Based on the feedback and experience gained in this R&D effort, WWT is poised to support internal and partner thermal camera deployments and offerings. We anticipate a significant number of organizations will be looking to install contactless temperature sensing solutions, and we will continue to advise customers and partners on how to be successful. From our initial development, we identified and documented several key takeaways.

The process of pixel transformation within images in a bi-modal camera setup is a significant challenge, especially for images with different aspect ratios. The team tried several methods, ranging from template matching to homographic transformations, which all worked for static images with the subject at a fixed depth. To mitigate this issue, the team used image space transformation with the cameras' intrinsic properties for a more dynamic approach to point translation.

While working on eye canthus detection, the team also devised a naïve approach which can be applied to IR images without much processing. Looking at the highest grayscale value in the IR image most often will point toward one of the two eye canthi. Figure 6 shows an example of the detection using this approach. The approach however can also result in false detections if a person is holding a hot object in the frame.

*Figure 6: Naive approach for eye canthus detection*

Finally, open source "face-landmark-detection" algorithms are robust enough, but sometimes the algorithms fail to adjust to evolving scenarios. As people started wearing face masks, initial results produced errors that necessitated a more robust algorithm.

In the future, as the team tries to detect facial landmarks using IR images, detecting people with a covered face will be an important evaluation criterion.

Closing thoughts

The team started with a scientific approach to measure the eye canthus temperature for the fever detection system. They included an optical camera in the setup due to the lack of availability of robust facial feature detection models on IR images. Once the eye canthus detection was achieved using the optical camera, the next challenge was to transform the detected point from RGB to IR image.

The knowledge of transformation using cameras' intrinsic and extrinsic properties helped the team. While working with a dual camera setup, the team also realized that extra caution must be taken to deal with synchronization issues between camera feeds.

After deployment, the team eliminated real-world challenges related to feature detection and temperature measurements using models from OpenVINO toolkit. The custom model performed well on people wearing masks, and since eyeglasses prevent accurate thermal readings, the system was enhanced to detect this and alert the system operator.

This prototype demonstrates a promising approach towards a cost effective and flexible temperature screening solution. Performing all data processing on the local computer allows the system to operate completely offline, supporting standalone deployment and potentially avoiding HIPAA concerns. Other systems can be integrated as needed to provide screening statistics or drive entry control hardware.

As businesses start to reopen, this will be a growing market. These systems would have limitations if they miss a single person with high temperature or if they have false detections. So the team focused solely on eliminating these issues. The experience gained from different challenges faced throughout this initiative will help WWT collaborate with customers to create a safer work environment.