Softbank's "SureTalk" that connects hearing-impaired and hearing-impaired people-I heard the secret story of development

What is "SureTalk" that connects hearing-impaired and hearing-impaired people?

The hint of service development was familiar

First, let's start with an overview of the service. "SureTalk" is a web tool aimed at mutual communication between sign language and voice. AI (artificial intelligence) tracks the body movements of sign language, recognizes the corresponding Japanese from the characteristics of the movements, and converts them into text. On the other hand, the voice uttered by a normal hearing person is converted into text by voice recognition and transmitted to the hearing impaired. The University of Electro-Communications is cooperating in the development.

Easy to use. Simply log in to "SureTalk" from your PC or iPad browser, click "Start Conversation", select the language mode (sign language / voice) to use, and enter the talk room to start a one-on-one conversation. It is a specification that can be used. We started providing services in the spring of 2021.

Converting sign language video into text

Mr. Takayuki Tanaka of Softbank Technology Management Headquarters recalls that hints for service development were hidden in familiar places.

"Now we are actively hiring people with disabilities, and we have also assigned hearing-impaired employees to my department. There is nothing wrong with them if it is a conversation between individuals. But 3-4 people At the meeting, the topic sometimes went on and on while waiting for a response. I used to use transcription tools, but as I read the text, the topic moved on to the next. We couldn't communicate well and sometimes frustrated each other. So, in order to develop a system that allows real-time conversation in handwriting, we applied for an in-house new idea recruitment competition and launched a project. That's how it happened. "

During the development, Mr. Tanaka was faced with the difficulty of Japanese again.

"For example, in Japanese, one word may have multiple meanings, or the meaning may change depending on the intonation. The same is true for sign language, and there are words that have multiple meanings in one expression. We usually infer meaning from context, but AI needs a "huge amount of data" to do this. "

Therefore, for the purpose of improving the recognition accuracy of SureTalk, we have prepared a function that allows general users to register sign language words and sentences (through a PC or iPhone). This is an attempt to widely collect data for AI to learn from users.

Anyone can register by imitating the movement while watching the sample video. Mr. Tanaka called for "I want everyone to grow SureTalk together."

Motion recognition by AI

Here, a small question. Is it okay for someone unfamiliar with sign language to register?

"Even if you think that you are doing the same movement, some people have a habit of doing it. If a large amount of data is accumulated by this registration function, the system will be able to absorb the error of such movement."

Aiming to expand the introduction destination while collecting data

About 10,000 words are currently registered. However, since it is designed so that it can be used in the system after a certain amount of data has been accumulated, it seems that the words that can actually be used in the system are narrowed down a little more.

"Currently, we are intensively registering words that are frequently used in local governments. In the future, we will add them as needed according to the usage scene."

By the way, there are two types of sign language used in Japan: "Japanese Sign Language" and "Signed Japanese". Currently, it supports "Japanese Sign Language", but in the future we would like to support "Japanese Sign Language", which is a complicated and difficult language for machines to learn.

The issue we are aware of now is that there is a limit to the sign language speed that the current system can catch up with. For example, we cannot keep up with the sign language speed that sign language interpreters do in sign language news on TV. Therefore, slow and clear sign language is required when using it. In addition, "confusion" occurs during similar sign language movements. It was said that this area will be resolved if data is accumulated.

Currently, we are expanding to local governments, but in the future, Mr. Tanaka hopes to expand to hospitals, hotels, public works such as railways and airports, and service industries. Dreams will spread.

Regarding the reaction from users, "We have received a very favorable response to our efforts. We would like to ask the general public to continue to cooperate with us, saying," Let's create a service together. " However, in addition to calling through our own media, I think that one day at SoftBank Shop, we will have the opportunity to explain to our customers. "

Kentaro Kondo

Born in 1977, graduated from Waseda University. After working for a publishing company, he became independent as a freelancer. He is active mainly in the telecommunications and digital industries, and has recently appeared in the sports and healthcare fields. He also loves interviews all over Japan and far away. His hobbies include camera, travel and playing musical instruments. He has also enjoyed shooting and editing videos.