Celemony’s Melodyne plug-in could already perform incredible feats of pitch manipulation with audio. But monophonic audio is one thing. Polyphonic audio has long been sound manipulation’s final frontier. With Melodyne 2, it seems Celemony’s audio wizards have finally cracked the problem.
Celemony is showing their new technology at Musikmesse, and they’ve got demos online you can check out:
Grab a note inside a chord, and you can manipulate that note directly. Retune it, change timing, adjust formants, change amplitude — timbre, time, and pitch are all accessible. Celemony is largely pushing this as a corrective tool, as that’s an obvious market, but needless to say, creative applications — even creative abuse — become interesting, too.
Melodyne Studio costs US$399 (349 EUR), with various discounts for upgraders, and the technology will be making more limited appearances elsewhere in Celemony’s product line. Now, it is a plug-in — clearly, someday this sort of thing will just be integrated directly in your host of choice, and I’m particularly excited about the day when it becomes a live performance tool. But for now, it could well be worth the cost of ownership.
You’ll have to wait a bit: the new version is scheduled to ship in the fall, though if you buy now, you’ll get the update free. Celemony, I’ll be seeing you at AES, I think.
Compatibility: Mac (Intel/PowerPC), Windows (XP/Vista)
Thanks to everyone who sent this in (Alex, Karsten, Eric, and others)! By popular demand, the demo video SonicState grabbed at Messe, because they’re organized enough to actually be in Frankfurt while I chill out here in NYC:
MESSE08:Melodyne Blows Our Minds
I hope to see this applied to midi guitar technology as well as sampling / synthesis in general.
I wonder if / when they will start to license it?
Well, you actually don't *need* a MIDI guitar with this. You could record a guitar, and then change the recording of what you just played as if it were MIDI, but of course with the full sound content of that recording and all the nuance that contains, which MIDI wouldn't provide.
License — I hope so. Perhaps as iZotope has done with a lot of their tech and other DAWs, we'll see "powered by Celemony" features in Logic, SONAR, Live, etc.
Check out the video Sonicstate posted of their presentation at Messe, it's even more convincing than the stuff on the Celemony site.
I use their current plug-in… Less as a corrective tool, but more for remixing things. Changing a melody to work with a new key or modality is pretty powerful. 2.0 looks even better.
Both evil and awesome, like all things that are truly awesome.
"But think about it, this is not meaty."
also, on a more valuable tip, i want this! Can't wait to see what I can do with it.
I wonder how the artifacts introduce compare to time stretching – i.e. if you stay pretty conservative its barely noticeable, if you crank it it starts to sound bad (which can be good at times of course)…
Is it comparable in that if I move it a minor third its less artifact-y than if I move it 2 octaves?
It would be fun to experiment with changing minor chords to major and vice-versa.
It'll really impress me if it can pick out notes that are only a semitone apart, playing at the same time.
Okay, I just watched the video, and now I feel like my last comment was redundant. 🙂
i am skeptical that it will sound this good. but i also don't think they would fudge a demo like that. could they be that amazing at developing software? this is really throwing me for a loop! i'll have to check out the sonicstate video as steve suggests
Amazing. His upcoming invention is direct transduction of thought to musical score.
absolutely insane this is…. It is a completely new way of working with audio in my opinion.
Regarding abuse, it will be interesting to see how this will work in sample-based music…. if an hip-hop artist change all the chords, and tweek the melody of some famous old recording and use it as a hook, is it still the same piece? How different has it got to be before they can get away with it?
OMG!!! It's the ultimate transcription machine! If you can't get a filmscoring gig after buying this software, you should just quit. All the secrets will be revealed. Those mysterious John Williams chord voicings will no longer be a mystery. This new Melodyne will be of enormous value to composers and music students. Yay!!!
I can't load the video now, perhaps it's slashdotted, so this is an uninformed comment, but … call me skeptical …
I'd imagine you'd have to feed this a *very* dry signal to get it to work.
Maybe I'm overly skeptical, but I can't imagine feeding a big complex signal like an orchestra, with all the overtones and reflections and reverbs into a machine and having it spit out the individual notes … and then letting me CHANGE them … AND having the result not sound like crap.
I CAN imagine feeding an unprocessed guitar into it and getting notes out. But I still can't imagine changing a note and having it not sound like crap … at least not in a complex setting. Maybe if you were real careful on your chord changes and didn't leave any squeaks or ringing strings … maybe.
quoted from Jamie K and the paris news group…
"I can imagine the sessions:
"Hal, we need you to strum a chord"
"Just one chord?"
"Just one chord, Hal, that's right."
"Uh, what chord?"
"Doesn't matter."
"Like this?"
"Great, thanks."
"OK, here I go…"
"No, we're done, Hal. You can leave now."
"Uh, really? OK, see you next time."
"We won't be needing you any more Hal, we have the chord to work with."
"wha…?"
"Hal, please clear the studio, we have the bass note to record, the horn
section stab, and then we need the rest of the week to work with your
chord."
"But I blocked out four hours…"
"Sorry Hal, we're only paying you for thirty seconds. But listen, if you
play that same chord on the piano on your way out we'll double your pay
and give you a soda. Which is like tripling your pay!"
"Yo man, you're a real jerk!"
"Nice, we can use the 'yo.' Thanks Hal."
"Mother#*&@ ****&#(#^!"
"Thank you. goodbye Hal."
For all the skeptics, we actually don't have to speculate much — Celemony already had Melodyne, and I've tried it. It works as advertised. I do expect a cluttered polyphonic recording could cause problems, i.e., you will need a clean recording. But it makes sense that it'd be possible theoretically — ever looked at overlapping overtones of different musical parts? On a spectral view, you can see how they relate to the fundamental in most timbres, if well recorded. So it's a math problem. Not an *easy* one, but not an impossible one — and Celemony's done a great job, it appears, building that into the interface. I'll make sure I'm on the list to test it in the fall.
@c64: lol.
Oh god this makes me moist.
Peter Neubäcker says on the Sonicstate Messe video that large reverbs and dense mixes don't really cause any problems. What does make it go crazy is clusters of swooping or decending notes. It's all discussed at the end of the sonicstate video anyway.
at 12 minutes in you can see / hear him pull apart a 1950s Chet Baker recording of a whole band.
What I'd like to know is if after a piece of music processed to reveal its note by note content, does Melodyne 2.0 also provide a look at the material in standard notation? It's not essential since melodic, harmonic, and rhythmic information is already available in Melodyne's "regular" view, but it would be cool to be able to view the information in standard notation.
[…] to correct wacky singing? You wish! Once again, much more and not less euphoric about it on CDM and MusicThing – and the press talk on Sonic […]
SIngers now has to start singing with huge glissandos and chinese opera-style vibrato to protect their work from this monster.
well, from a technical point of view:
– most instruments show a strong base frequency at the basic note, and harmonics with an odd and/or even whole-number multiply factor
– this means if you do a fine FFT and start to "Look" from the lowest frequency upwards, the first strong discrete freqency should be lowest note in the chord
– you mulitply by 2/3/4/5… and you find all the harmonics releated to this lowest note
– you move on from this first tone, and if you hit the next strong frequency which is NOT an odd/even multiply of the first, you have got the next note in a chord …. and so on…
with this technique, it is plausible that you can separate all the notes from a cluster of 12 half-notes within one octave, played at the same time. if (for the sake of easy explanation) the base note is 100Hz, the corresponding harmonics are at 200, 300, 400, 500Hz etc… the next half-tone above this is 100Hz times 12throot of 2, so it is 105,94Hz, which can be discriminated from 100Hz with a good FFT. so even from this very complex chord, the 12 notes with their harmonics can be clearly identified.
I think the only problems might be OCTAVES played simultaneousely (as those are precisely double the frequency)and the non-harmonic noises which are important for some instruments.
fascinating processing anyway.
@ karl
you may wish to do actual experiments to conclude if your claim that harmonics fall at exact integer multiples is true.
the problem is a bit more tricky.
[…] This is Peter Neubäcker from Celemony. You’ve probably already seen this promo video of his new product, Direct Note Access. It’s a new version of the autotune-type pitch correction software which – it appears – can work with polyphonic sound. Record a chord, and it lets you explode that chord and re-tune individual notes. I thought that this was impossible. Peter Neubäcker says “What doesn’t work in theory can still work in reality.” Well, maybe. In May 2005, a startup called Zenph Studios claimed to have cracked the problem of polyphonic transcription. They analyse old piano recordings (i.e. Glenn Gould playing Goldberg Variations in 1955) and produce a high-resolution MIDI-type file with exact pedal movements and note/pressure data. They feed that into a Disklavier MIDI grand piano, and record the results. They’ve had good reviews (at least in audiophile mags) for the recordings. The potential of this kind of polyphonic transcription is enormous – it would let you sample a performance, not just the recording of a performance. Zenph may be able to do it in a slow, precise, way – presumably with a considerable amount of human help, and they’re just pulling out note data, not separating the actual sounds of the notes. Celemony are claiming a lot more. If it works, it’s a revolution. It shouldn’t be long before you can separate any mixed recording into unmixed tracks. You’ll be able to turn any guitar into a guitar synth with no special hardware. It’s very exciting. Does it actually work? I can’t imagine how it could, but I know almost nothing about signal processing or the theory of sound. That’s where you come in… (More coverage at Create Digital Music) […]
Sorry if the video has been a little clunky, we've been rather snowed under this messe. But I was at the demo and it was pretty amazing. I suspect that the audio examples were well chosen, but they were diverse for sure.
It was kind of like a Steve Jobs keynote speech – cheers and woops etc. I would love to listen in a more sterile environment for sure, but either way it has enormous implications for music production.
I would think that it will work pretty well, I just cant see Peter Neubaker trying to fool us – he's a pretty straight guy
I have never had such a feeling of awe at a piece of software before.
Hardware yes, but this?
It's like the Guy has invented Sampling All Over Again just like Sonic Foundry Acid changed looping from something of an art to an auto response.
all thats left now is turning a full multi-instrumental recording into a series of midi and sound samples and i'd be in heaven. Can't imagining it happening any time soon though. 😉
[…] Vanitha wrote an interesting post today onHere’s a quick excerptGrab a note inside a chord, and you can manipulate that note directly. Retune it, change timing, adjust formants, change amplitude — timbre, time, and pitch are all accessible. Celemony is largely pushing this as a corrective tool, … […]
[…] This is Peter Neubäcker from Celemony. You’ve probably already seen this promo video of his new product, Direct Note Access. It’s a new version of the autotune-type pitch correction software which – it appears – can work with polyphonic sound. Record a chord, and it lets you explode that chord and re-tune individual notes. I thought that this was impossible. Peter Neubäcker says “What doesn’t work in theory can still work in reality.” Well, maybe. In May 2005, a startup called Zenph Studios claimed to have cracked the problem of polyphonic transcription. They analyse old piano recordings (i.e. Glenn Gould playing Goldberg Variations in 1955) and produce a high-resolution MIDI-type file with exact pedal movements and note/pressure data. They feed that into a Disklavier MIDI grand piano, and record the results. They’ve had good reviews (at least in audiophile mags) for the recordings. The potential of this kind of polyphonic transcription is enormous – it would let you sample a performance, not just the recording of a performance. Zenph may be able to do it in a slow, precise, way – presumably with a considerable amount of human help, and they’re just pulling out note data, not separating the actual sounds of the notes. Celemony are claiming a lot more. If it works, it’s a revolution. It shouldn’t be long before you can separate any mixed recording into unmixed tracks. You’ll be able to turn any guitar into a guitar synth with no special hardware. It’s very exciting. Does it actually work? I can’t imagine how it could, but I know almost nothing about signal processing or the theory of sound. That’s where you come in… (More coverage at Create Digital Music) […]
I see many comments here about MIDI who cares about MIDI it has always sounded crap anyway do these people not understand what this man has done these are AUDIO files here real sounds data played by real people and they are being used like Midi Files I dont realy think this Achievement has sunk in yet for some people , it has for me being a guitar player ,in brief i think this will go down as one of the greatest achievements in music history ———
Just as a finale note to Peter Neubacker i will forgive you for putting me out of a good job as one of the best Guitar Players in England if you send me a free copy of DNA lol
and once again congratulations
wow this program is amazing