You may soon listen to YouTube videos in any language thanks to AI
Videos uploaded to Google's video hosting site YouTube are usually available in a single language. To make these available to YouTube users who do not understand that language, Google introduced subtitle options to YouTube a long time ago.
Nowadays, captions may be generated automatically on YouTube. While the automatic generation of captions works so-so, it still makes content accessible to users who would not be able to make use of it otherwise.
Soon, YouTube users may listen to videos in languages that they understand, even if the original language of the video is not among them. Powered by Aloud, a Google Area 120 project, videos may be dubbed automatically soon on YouTube using AI.
Aloud was first unveiled in 2022 as an Area 120 project. Its main purpose back then was the creation of tools to help video producers create subtitles for their videos in different languages and dubbing. Dubbing refers to a video production technique that changes the language of video by swapping audio and trying to match the new audio track as closely as possible with lip activity.
Aloud does most of the heavy lifting for video producers. It transcribes the video in the beginning and the video producers may review and edit the transcription. Once satisfied with the result, Aloud translates the text into selected languages and produces the dub.
Aloud published a short demonstration video last year that highlights how it works:
Aloud is in early access currently; this restricts access to the technology but also the languages that it supports at the time. Only a few languages, including English, Spanish or Portuguese are supported at the time.
YouTube appears to be testing the tool currently with some creators, according to a statement by Amjad Hanif, Vice President of Creator Products, YouTube.
Interested users and content producers may check out the following video on YouTube. Its original language is English, but it is also available in Spanish thanks to Aloud.
Select Settings > Audio Track > Spanish (Latin America) dubbed to switch to Spanish.
YouTube has big plans for Aloud. Besides extending support to other languages, Aloud will have capabilities to mimic the creator's voice in the target language, improve lip syncing and more expression.
Closing Words
Applications for such a technology are far reaching. YouTube content producers may use it to reach an audience that might otherwise be out of reach for them. It remains to be seen how well the transcription and creation of foreign language audio works, and how much work producers need to invest into creating good transcriptions that the AI may use to create the additional audio tracks for their videos.
Now You: What do you prefer, subtitles or dubs?
Never buy an English language DVD or BluRay disc in Germany: they dub everything!
Aside from that though I prefer subtitles since I prefer to hear the original recording in whatever language it happens to be. BBC News dubs all their recordings which is really annoying because you can still hear the original language in the background with the dubbing played over the top of it.
DVDs and BluRays in English but available in the Netherlands (and presumably other languages as well) have Dutch subs burned in which is equally annoying since you can’t turn them off.
So I buy all my DVDs on Amazon UK. At least that way I know I’ll be getting the original language versions.
What do I prefer, subtitles or dubs?
Subtitles, definitely, be it for videos as for movies.
If I understand the original language, obviously (in which case I disable the subtitling if applicable) and if I don’t I nevertheless prefer to understand only with subtitles and not miss the human’s tone, accent, I mean every component of a speech “musicality” so to say, but I admit this is eminently subjective.
By the way, in the article’s video, the lady speaking in Bahasa-Indonesian was obviously post-synchronized; what a pity I no longer read on the lips as I poorly did at one time.
I can imagine this being used for spreading propaganda like it can be with subtitles – if the viewers don’t speak the language and haven’t the slightest clue of it, you can give them any translation and they can’t really tell if it’s true or a lie.
It can be a video about herding cattle and the subtitles or dubbing can be talking about the poor rural life in 3rd world country, nobody’s going to know the difference.
It’s made for people who don’t question anything and aren’t inquisitive in their search for the truth, the actual truth, not the propaganda by state media.
This is scary and a concern, but then again YouTube has become a garbage website in the past 4-5 years, I’m finding Rumble to be a better alternative, the only problem is it lacks content and creators, but there are some very interesting videos about the truth in life, videos that YouTube would probably ban and censure.
@Ty:
“no more language barriers. Truly momentous game changer for the human race!”
So was the Tower of Babel:
https://en.wikipedia.org/wiki/Tower_of_Babel
Half of them are incomprehensible in any language.
My native language is Spanish and I can tell you that that Amoeba Sisters video is quite impressive. It is indeed a “neutral” Latin American Spanish accent (as opposed to Spanish from Spain). It reminds me a bit of the dubbed audio from TV cartoons
You can tell it’s been machine generated, in the sense that it sounds slightly robotic, but otherwise it’s really good
It was a matter of time; next will be real time augmented reality with head set, translating people, images, and videos around you while you travel… no more language barriers. Truly momentous game changer for the human race!
This sounds great. I often look at things such as product reviews or maintenance instructions on YouTube, however quite often if the product is fairly specific, then there may not be many videos in English. Therefore, being able to understand German or Dutch videos for example would be great – it would mean not having to just look at the visuals and try to imply what they’re saying based on their body language and tone.
Unrelated to the article, but another useful YouTube feature is clicking on the three dots ellipsis button (…) and going to “Show Transcript”. It’s then possible to use the browser “Find on Page” search feature to find particular words in the video. For example, you may have previously watched a two hour long video, but want to re-watch a particular part where they talked about “Shimano” – you can us the transcript to search for “Shimano” and clicking on it will take you to that exact part of the video.
“What do you prefer, subtitles or dubs?”
I always have subtitles on, even in my preferred language because audio quality and actor/actress enunciation can be frustratingly incomprehensible.
“Dubs?” The last time I inadvertently started watching something that was dubbed, it was an immediate turn-off. May have been the show, but it was too obvious. Really poor quality.
I think I would have to be stuck in a hotel room in Cambodia during monsoon before ever trying “dubs” again.
This is one of the most interesting news of this year. Absolutely good idea!