Speech to text —
Book publishers sue Audible to stop new speech-to-text feature
Publishers say Audible's new captions feature is illegal. Is it?
Timothy B. Lee - Aug 24, 2019 11:55 am UTC
with 84 posters participating, including story author
Seven of the nation's top book publishers sued Amazon subsidiary Audible on Friday, asking federal courts to block the company from releasing a new feature called Audible Captions that's due out next month. The technology does exactly what it sounds like: display text captions on the screen of your phone or tablet as the corresponding words are read in the audio file.
The publishers argue that this is straight-up copyright infringement. In their view, the law gives them the right to control the distribution of their books in different formats. Audio is a different format from text, they reason, so Audible needs a separate license.
This would be a slam-dunk argument if Audible were generating PDFs of entire books and distributing them to customers alongside the audio files. But what Audible is actually doing is subtly different—in a way that could provide the company with firm legal ground to stand on.
The caption feature "is not and was never intended to be a book," Audible explained in an online statement following the lawsuit. "Listeners cannot read at their own pace or flip through pages as they could with a print book or eBook." Instead, the purpose is to allow "listeners to follow along with a few lines of machine-generated text as they listen to the audio performance."
"We disagree with the claims that this violates any rights and look forward to working with publishers and members of the professional creative community to help them better understand the educational and accessibility benefits of this innovation," Audible added.
Audible's case likely rests on two landmark video recording precedents
hcalderonmeister
Audible hasn't fully explained how Audible Captions work, but the company has said enough to provide a pretty clear picture. In a video about the service released earlier this month, an Audible executive explained that the technology was "built on publicly available technology through AWS Transcribe." That's Amazon's cloud-based service for automatic text transcription.
So it seems that the Audible app is generating text captions in realtime as the user plays an audio file. The app sends snippets of audio files to an Amazon server and gets back corresponding sections of text, which it then displays on the screen one word at a time. (It's possible that AWS Transcribe has an offline mode that allows the transcription to happen on-device, but I haven't found any documentation about this. I've asked Audible about this and will update if they respond.)
Audible is likely doing this because it strengthens the company's argument that it can do this without a license from publishers.
To see why, it's helpful to review two of the most important copyright decisions of the modern era. The first was the 1984 decision of Sony v. Universal that declared the VCR legal. Hollywood argued that the "record" button on a VCR was an invitation for customers to infringe their copyrights. But the Supreme Court disagreed, arguing that copyright's fair use doctrine allowed "time shifting"—recording a show now to play it later.
The courts built on this decision with a 2008 ruling known as Cartoon Network v. Cablevision. In that case, a bunch of media companies sued the cable company Cablevision because it was offering customers a "remote DVR." Like a conventional DVR (or a VCR before that), Cablevision's technology allowed customers to record and play back television shows at their convenience. But unlike a conventional DVR, the remote DVR was located in a Cablevision data center, not in the customer's home.
Unlicensed: Are Google Music and Amazon Cloud Player illegal?
Television content owners argued that Cablevision was infringing their copyrights by making unauthorized copies of their show on a massive scale. Cablevision disagreed, arguing that the copies were being made by customers, not by Cablevision. The physical DVR might be owned and maintained by Cablevision, but the customer was deciding which shows to record. And the customer was entitled to do that under the earlier Sony ruling. An appeals court ultimately accepted this argument.
The Cablevision ruling provided a legal foundation for cloud-based "storage locker" services that allowed customers to upload, save, and stream (but not share) their music and video collections.
Expect Cablevision to be central to Audible's defense
That brings us back to Audible's new transcription technology. Audible doesn't have the legal right to sell text versions of audiobooks to customers without publishers' permission. But we can expect Audible to argue that it does have a right to sell software tools that allow customers to do speech-to-text conversion.
Audible's case will likely be strengthened by the fact that its app never creates or saves a permanent, full transcript of an audiobook. Instead, the software only displays a few words on the screen at a time.
If Audible is sending audio files to Amazon's servers for transcription, publishers are likely to argue this means Amazon—not users—are creating the transcripts. But this seems closely analogous to the Cablevision case: the conversion is being done by Amazon servers but only when explicitly requested by users. And each translation is only sent back to the user who requested it.
Of course, outcomes in disputes like this aren't easy to predict. Courts could decide that a conversion from audio to text doesn't have the same fair use protections as the time-shifting activities the courts have blessed in the past. The courts could decide that Amazon plays too active a role in the conversion process to portray itself as a passive supplier of technology like the maker of a VCR. Or they might decide that this whole legal argument is a too-clever-by-half ploy to justify reproducing publishers' copyrighted books. Ultimately, we'll have to wait and see what the courts say.
Interestingly, Amazon faced the same battle in reverse a decade ago when it proposed adding a text-to-speech feature to Kindle. Like Audible today, Amazon was planning to offer the text-to-speech feature for all e-books without getting approval from book publishers. But after a backlash from publishers, Amazon caved and gave publishers the right to disable the feature. Amazon now says that "most Kindle content and personal documents are eligible for Text-to-Speech."
Promoted Comments
J.King Ars Scholae Palatinae et Subscriptor
kris44dad wrote:
And, I don't think a deafness argument will work. The counter argument being, why would a person hard of hearing have a need to buy an audio book, in preference to the printed copy?
Most disabilities, including hearing loss, can be temporary. They can also occur after one has already purchased things like audiobooks.
937 posts | registered 9/10/2015
Crescent Smack-Fu Master, in training
xyzzy01 wrote:
show nested quotes
Being an Audible member since 2001, I look forward to this feature. So many times I hear something I don't understand and would like to see the passage in print without having to locate it in a text version.
One problem is that they are automatically transcribing the book. Names, technical terms, unusual words, and words not found in dictionaries at all will probably be butchered.
That's a good argument that publishers won't really be impacted by the close captioning of audio books: If you need a really accurate rendition of the audio book you'll buy a printed (e-book, etc.) copy.
Close captioning for audiobooks is much more of an ADA issue. Hearing issues aren't just volume issues which can be fixed by increasing the volume. Some people don't hear some sounds as sharply as others or they can't easily isolate the sounds they're interested in from background noise. Captioning, even if the occasionally word is misspelled, makes it so much easier to figure out what was actually said.
For people who are trying to improve their verbal comprehension skills in another language, captioning makes the audio book makes it more desirable. If I were learning a new language and had a choice between 2 audio books in that language - one with captioning and one without - I'd be far more likely to buy the captioned book. Either way, I probably wouldn't buy the print book on top of that.
This is a case of cutting off your nose to spite your face. Technically the publishers might lose a little in revenue in print sales from the captioning, but they could gain a much wider audience if they embraced the captioning.
10 posts | registered 12/15/2017
Timothy B. Lee Senior tech policy reporter
Bigdaddyguido wrote:
But what you're describing is a potential demographic, selling your text and audiobook bundled, which we do. Also, we'll give discount codes with the purchase of one towards another.
But, beyond all of that, the bigger issue here is that audible is attempting to disseminate an authors work in an entirely new medium without getting consent or cooperation from the creator. That's a dangerous precedent to set to say the least.
Why not just provide an option for the publisher to provide an accurate transcript which they can display with the audio? Why circumvent the artists intebtions and do this all artificially, with no controls on what gets translated and how?
Maybe I should have discussed this in my article, but Audible already does this with a product called "Immersion Reading." This product allows customers to buy both an audiobook and a licensed e-book, and Audible syncs them up so that each word in the text gets highlighted as it's read in the audio file. Customers can scan forward or backward through the text like you can with a regular e-book (which they can't do with captions).
Naturally, Audible and the publishers interpret this in opposite ways. The publishers argue that the existence of Immersion Reading shows that Audible has already conceded the principle that they need a separate license to show text alongside audio files—and that Audible Captions is basically a pirated version of Immersion Reading.
Audible, on the other hand, stresses that Immersion Reading and Audible Captions are different products aimed at different markets, and that most of the people who would have bought the Immersion Reading bundle would not be satisfied with Audible Captions. This thread someone illustrates the point. Because Immersion Reading is working with the original text, it comes with correctly spelled character names, foreign words, place names, etc. So if that's important to you you've gotta buy the licensed product—the unlicensed machine translation probably won't meet your needs.
1256 posts | registered 1/6/2007
Excors Wise, Aged Ars Veteran et Subscriptor
It would be fun for the lawyers if (hypothetically) the Amazon Transcribe machine learning algorithms had been originally trained using the Audible audiobooks and Kindle ebooks. That's millions of hours of high-quality speech recordings and corresponding text, which seems like great training material. As far as I'm aware, using copyrighted material for training is probably considered fair use in the US (although the law is largely untested and unclear so nobody really knows), and the machine learning model is not considered a derivative work of the training data. So Amazon could probably do that without permission from the publishers.
That means they could freely use the ML models that had been trained on copyrighted audiobooks and copyrighted ebooks, to transcribe exactly the same audiobooks back into text that looks a lot like the ebooks. The service might be able to perfectly transcribe a weird fantasy character name that was read in a strong accent, because the ML model was already trained with that specific name and it encoded the knowledge in some enormously complicated configuration of neuron weights. Should that be legal? It's not the same as just storing the copyrighted text in a database - the original data has been transformed and combined beyond all recognition. But it behaves a lot like that database.
Copyright law around machine learning seems likely to become a very messy topic in the near future.
Lendorien Smack-Fu Master, in training
As someone with hearing loss in certain registers, I fully understand the purpose of this feature. I watch TV with captions on because it increases my comprehension of what I'm listening to. I don't need it to hear and comprehend, but it does help.
Personally, I don't see how this hurts the publishing industry. Its not like the text is scrollable or available separately. You can't read it faster or slower than the audiobook speech. If and thing, this is a handicap assistance issue. Misshear some part of the audio, roll it back and look at the captions.
The publishers are entirely on the wrong side here. I get the concern, but this is very differant than say.... including a copy of the novel with the audiobook.
Amazon should make the ADA argument in their defense. Is this that different than providing subtitles for TV? It assists the hard of hearing have access to the medium.
chilinux Wise, Aged Ars Veteran
I hate to say this but I think there is a couple things that might work against Audible in this case:
(1) Licensing of the material to Audible with the understanding it's use would be controlled by Audible own proprietary Digital Rights Management. If Audible makes the case the customer is able to produce speech to text without ever having to re-sample the audio through a microphone then Audible may have failed stated obligations of the level of protection their DRM provides. They may also make the case the new feature violates the DMCA as acting as a copyright circumvention device even if the transcription is only momentary.
(2) Several people have pointed out that Audible should be able to appeal to the protections provided by the American Disabilities Act. The case of Access Now, Inc. vs Southwest Airlines showed that the courts seem to have a very narrow view on the scope of what the ADA provides protections for. In that case, the ruling even indicated that congress should consider amending the ADA to provide a more modern scope. The fact that congress has done very little to improve the ADA, it may be possible that while logically Audible should be able to use the ADA as a defense, the courts may still not see it that way.
Overall, I am extremely disappointed in how the concept of fair-use is continually eroding. I want to hope for the best but my experience has taught me to expect the worst.
Timothy B. Lee Timothy is a senior reporter covering tech policy, blockchain technologies and the future of transportation. He lives in Washington DC.
Email timothy.lee@arstechnica.com // Twitter @binarybits
Vinbullet Smack-Fu Master, in training et Subscriptor
reply Sat Aug 24, 2019 7:04 am
We need more companies like Amazon to stand up and fight for the rights of deaf people reading audio books /s. In all seriousness I can't see this being a bad thing if you can't scroll the text itself.
zsak23 Smack-Fu Master, in training et Subscriptor
88 posts | registered Jul 3, 2012
I'm entirely unmoved by the publishers' argument. A real-time sliding-frame transcript is not a published, separately available product, and is not a substitute for the printed book. It can also clearly be an accessibility feature, not to mention a useful tool for someone not entire proficient with the language of the audiobook.
Please stop trying to tell consumers what they can and cannot do with their computers*.
* I realize Amazon servers do seem to be involved here, but it doesn't seem like that's actually a necessary ingredient if the client isn't an anemic mobile e-reader.
KeyboardWeeb Ars Scholae Palatinae et Subscriptor
Copyright is nuts sometimes. Of course this should be fair use, especially considering the transcriptions are both tiny (representing an insignificant fraction of a work) and ephemeral--unlike the recording of last night's Steven Colbert that sits on my dad's Tivo right now.
Do the publishers seriously think this will hurt them in some way? Why? How?
999 posts | registered Sep 28, 2016
Marc GP Ars Centurion
The ability to read at the same time that you hear is invaluable to learn English. I usually do it with TV shows and English subtitles, but it's nice to have more options.
pauli Moderator et Subscriptor
Marc GP wrote:
It's a necessary thing for a lot of us native speakers, too.
34993 posts | registered Feb 23, 2000
Elektriktoad Seniorius Lurkius
If it's anything like the automatic speech to text transcription services I've seen, the publishers shouldn't have anything to worry about -- Audible's text version will be an amusing facsimile of the original with comically wrong proper names.
Rbrian Smack-Fu Master, in training
Although my hearing is ok, I struggle to focus on audio, and find written text far easier to process - I use subtitles on TV, though I'm not dependant on them. For this reason, I'd never consider an audio book - but with subtitles, I might. I'd still get the benefit of hearing the correct pronunciation of unfamiliar words, or unusual names, whilst having something to look at and anchor my focus.
Why would this upset the publishers? Do they expect me to purchase audio books AND a written version? Not gonna happen! I'd stick to the much cheaper text option, so they'd earn less money overall.
34 posts | registered Feb 10, 2016
Polykin Smack-Fu Master, in training et Subscriptor
zsak23 wrote:
I was thinking the same thing. I suspect however that the audio-to-text won't be any better at discerning difficult words than humans, and that anything I can't understand won't be understood by the transcribing software either, especially when it comes to unusual names.
Last edited by Polykin on Sat Aug 24, 2019 8:01 am
8 posts | registered Mar 6, 2018
PokemonPets Ars Centurion
This has no negative effect to publishers. I am expecting court to deny this copyright infringement claim
Polykin wrote:
I was thinking the same thing. I suspect however that the audio-to-text won't be any better at discerning difficult words than humans, and that anything I can't understand won't be understood by the transcribing software either, specially when it comes to unusual names.
You are correct, there are some books (ex, Bernard Cornwell's The Last Kingdom series. I listen to the audio and follow the text) where the spelling and pronunciation are radically different. But there are plenty of times where a narrator will be less than clear and a look at the text is enlightening.
Voyna i Mor Ars Scholae Palatinae et Subscriptor
KeyboardWeeb wrote:
Of course not. They see an opportunity to gain money.
5164 posts | registered Nov 3, 2015
Exactly. That is exactly why the description of the feature left me disappointed. I hoped it was essentially a "follow the audio with text directly from the book", or at least that any names could be pre-added to the subtitles to avoid speech-to-text mangling it. That publishes sue simply for a text-to-speech generator leaves me less than hopeful.
I love Kindle's and Amazon Prime Video's x-ray functionality in order to remember characters. That the Audible app doesn't have it (or at least not for any books I've listened to) is unfortunate.
alex_d Ars Scholae Palatinae
Copyright owners should not have the right to control the way a work is rendered. They should be happy to make the $10 or whatever fee that they now earn per book and have it cover audiobooks, performances, movies, etc and shut the fuck up. (Obviously such a fee would be too high for a movie, and movie studios would continue to negotiate movie rights, but the option to skip negotiations would be there.)
Copyright owners deserve far less power.
Last edited by alex_d on Sat Aug 24, 2019 8:16 am
1095 posts | registered Mar 6, 2010
Schwanz Ars Praefectus et Subscriptor
I'm listening to an audiobook right now where the narrator has a fairly strong accent, and there are a number of words I still haven't nailed down despite hearing them often. This feature would be great.
Although, now that I read the article, it seems that they're just going to be doing the same sort of machine transcription that utterly fails to properly caption youtube videos. -ah well-
Causality Ars Tribunus Militum
Same. My usual habit is to read the first chapter of a book before continuing with the audio version because I want to get a good handle on the character and place names. It's jarring to realize 90% of the way through you've been hearing the protagonist's name wrong.
Sirambrose Smack-Fu Master, in training et Subscriptor
This will be useful when listening to books written or narrated by someone from another country. I often need to skip back and re-listen to a sentence when i listen to books from England, especially when words or expressions not used in the US are mixed with made up words from the fictional setting.
What I would really like is an index for audiobooks. For some reason, audio books usually number the chapters even when the book had chapter names. I usually don't remember the chapter number of a section of the book I want to skip back to. For books that switch between characters, it would be useful to skip backwards to the last time a character was mentioned to refresh my memory of what they were doing.
harmless Ars Tribunus Militum et Subscriptor
So, what if I'm doing the conversion on my own? Is that copyright infringement? And how is anyone supposed to find out? Having protections that are not enforceable is not a good idea.
And if I am allowed to do a transcription for myself, why would it be illegal for Amazon to do that on my behalf?
In conclusion, I think having separate licenses for different formats of the same work is a stupid idea.
slopoke Ars Centurion et Subscriptor
This is nothing more than closed captioning. That has been extended (from TV to You Tube to video games) every time there's been a new format. I expect this will go the same way. If the courts decide against Audible then the law will get changed. The book publishers should quit now and save themselves a lot of expense.
Bigdaddyguido Smack-Fu Master, in training
slopoke wrote:
This is a great point. Question, tho. Are CCs created by the distributor or the producer? Like, am I allowed to create closed captions for someone else's work that I have the right to distribute?
Edit - The more I think on it, the more I think this stumbles on something here. When you watch something with closed captions, it gives credit to the creator with something like "closed captions are provided to you by".
Judging by the comments, one of the major desires here by users is to understand text which is muffled or disguised by an accent, etc. These issues won't be resolved by a speech-to-text algorithm, and often, could be exasperated, with the converter giving incorrect translations.
It seems like a much better solution is for Audible to work with publishers to license an actual transcript from the publisher which they can provide in time with the audio, CC style. This way everyone wins and the actual desires of the end users are achieved without blindly circumventing the creators intentions with their work.
Last edited by Bigdaddyguido on Sat Aug 24, 2019 9:42 am
82 posts | registered Mar 21, 2008
irnoob Wise, Aged Ars Veteran
ADA versus the sundry copyright laws. Which is controlling?
562 posts | registered May 23, 2019
Demento Ars Legatus Legionis
Somehow the publishers have managed to make Amazon - a multi-billion dollar behemoth, infamous for dodging tax and poor treatment of their employees - look like the good guy.
I would argue that the only groups capable of doing so are the publishing and film industries. It just shows how sheltered and insular they've been over the years.
12326 posts | registered May 12, 1999
LiKenun Ars Centurion
pauli wrote:
I can't recount how many times I've relied on closed captioning. When a character on TV mumbles, says something with a funny accent, or just speaks in blurred dribbles, I don't want to (or can't) rewind and replay to get a better handle on what was said.
panton41 Ars Praefectus et Subscriptor
LiKenun wrote:
Between bad hearing* and only half paying attention closed captions has made watching TV a lot easier. First time I realized I needed it I was watching the David Lynch Dune with my ex girlfriend and first noticed Dr. Kynes saying a prayer for the passing of a Maker and suddenly the character took on a lot more depth.
* I play my music loud because I can't hear it and I can't hear it because I play my music loud.
NotContinuum Ars Legatus Legionis et Subscriptor
If several people requested Cablevision record a program, could the service have just one copy of the program to save HDD space and let those individuals have access to it? Or would that be considered distribution and illegal?
Downrazor Smack-Fu Master, in training
Look, how is this any different than having a PDF that gets read out loud by the PDF reader program?
This already exists and takes one medium (text) and transforms it to another (speech) to deliver content to the end-user.
The Audible mechanism would only do this process the other way around (Speech to Text), still transforming the medium, and this is almost identical to the closed-captioning that is required for many television shows..
I don't see how there is a legal problem..
81 posts | registered Sep 26, 2016
Operative Me Ars Legatus Legionis et Subscriptor
reply Sat Aug 24, 2019 10:07 am
Letterkenny is so much more enjoyable when I can fully parse what is being said.
xyzzy01 Wise, Aged Ars Veteran et Subscriptor
Ozmodan Ars Scholae Palatinae
As a book publisher, a lot of work goes into cresting a physical and audiobook copy of the book. For education titles, yes, there is an expectation that the same user could want to buy a physical book and an audiobook as they each serve different purposes in the learning process.
I understand that what audible is doing is different than just giving them a pdf of the book, but for non-fiction in particular, this does seem like it could be infringing on the authors and publishers ability to properly sell their book. Not saying it is, but that it's not this clear slam dunk the comments seem to imply.
Also, people are saying this will be useful for understanding muffled audio. Since when is speech-to-text generation better at understanding human dialog than humans? The only way this would be able to decipher unintelligible audio would be if it's referring the audio back to a master text file that was produced by a human. I don't think that's audibles plan here, but if it is, that seems a bit different than what the article described.
Your argument is ridiculous, the publishers are already getting money from Amazon for the audiobook, what you are condoning is double dipping. I hope Amazon destroys this idiotic attempt to extract more money from them. Just about one of the dumbest copyright suites I have seen.
818 posts | registered Sep 9, 2008
I wonder how much of this is a cash grab attempt by the publishers vs. them simply investing in keeping their reputation of vigorous copyright protection intact. Probably a whole lotta A and a little bit of B.
I understand that what audible is doing is different than just giving them a pdf of the book, but for non-fiction in particular, this does seem like it could be infringing on the authors and publishers ability to properly sell their book.
Do you have sales figures showing how many people buy both the audio book and the digital book? The audio book versions are generally more expensive than the kindle versions of books in the first place, so we can pretty much eliminate the "I'm just going to buy the audio book version and read the transcription mwahahahahah" theory out of the gate.
So the only people who might be influenced not to buy the book itself are people who are already buying the Audible version. How many people is that? What's the conversion rate? Compare that to the potential audience increase from providing a superior product.
Hell, I'll tell you the truth, my interest in audio books is near zilch because I'd have to buy both. Make it a package deal for a reasonable price and you'd actually get a hell of a lot more revenue from me. I like the notion of audio books, but I'm not buying both separately. Ever. Waste of my money.
So 99.99999% of the time, I'm getting the kindle version. This transcription service actually might make me consider buying an audio book instead. You're looking at the potential for a sale you never would have had without it.
But hey, ignore what customers like. I'm sure that's a good strategy.
iAPX Ars Centurion
Many computers and devices have speech-to-text technology (usually very bad ones, but still), so you could play an Audio Book in front of your computer, tablet or smartphone and theoretically have the transcribed text as a result.
Will they sue any provider of these solutions?!?
ColdWetDog Ars Tribunus Militum et Subscriptor
Elektriktoad wrote:
Isn't parody a copyright defense?
Could work for Amazon as one of those 'Hail Mary' arguments.