Show HN: I trained a 125M model to autocomplete piano on-device

Posted by simedw 9 hours ago

Show HN: I trained a 125M model to autocomplete piano on-device(simedw.com)

I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15).

The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device.

The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.

421 points | 98 commentspage 2

devonsolomon 8 hours ago|

This is really fun. Scaler 3 starts with a chord progression and lets you break it down into musical performances and parts. Useful for ideation when producing.

Would be fun to get a midi clock going and play some chords on my piano and have my synth start jamming along with the bass and my keyboard doing some performance. Or any combination of the above.

mh- 7 hours ago||

+1 all of this. That would be incredible (this already seems very cool - excited to get home and try it!)

Naitik88 6 hours ago||

That would be a really interesting direction. At that point it starts feeling less like autocomplete and more like having another musician reacting to what you're playing in real time.

rajivayyangar 3 hours ago||

The results strike me as comparable or worse than you could get with a Markov model. I think it reveals the gap in understanding between LLMs and music. I think you need to either: - Set up a pipeline to decompose music into, say, harmonic sequences and melodic sequences, and then have the LLM work on some more fundamental or more high-level layer of musical composition and then re-translate it back into actual sounds. - Develop a better dataset and train the LLM more natively on musical examples.

Does anybody know of a project that has produced more convincing results?

Computer0 41 minutes ago|

There is no LLM in the ML pipeline provided by the OP.

sagebird 6 hours ago||

Für Elsevier Journal Access Library pass out candy for safety this halloween your baby off milk shake it off.

elwell 5 hours ago|

I, für one, welcome our new LLM overlords.

axoltl 2 hours ago||

Looks neat! I'd love for it to feed the MIDI notes back into my player piano instead of playing out of my iPhone's (comparatively tinny) speakers though.

evalystai 8 hours ago||

The idea is awesome! :) However there's definitely much room for improvement, first of all rythm and composition (so there's some sense of musical form).

simedw 8 hours ago|

Thank you.

Yes, I think I’ve gotten it to roughly a GPT-2 level: good enough to share, but with a lot of room left to improve. I think adding some kind of bar/measure token might help with rhythm, and perhaps some form of longer-term planning for the overall composition.

bravura 4 hours ago||

You could also distill the fabulous Anticipatory Music Transformer from Stanford. https://crfm.stanford.edu/2023/06/16/anticipatory-music-tran...

Also consider checking their decisions about representation, etc

Tepix 6 hours ago||

Oh, a cool idea! I just tried it, works pretty well. Kudos!

One feature request:

Instead of playing the AI-generated audio solely through the iPhone's speakers, add an option to send the audio as midi notes to a device (probably the same one you received the mini notes from).

Ohentis 3 hours ago||

It occurres to me that the relationship between pitches usually matters more than the pitches themselves. Perhaps the pitches could be encoded in a vector like what is typically done for position.

butlike 5 hours ago||

Question as a musician: Do you find satisfaction in the piece it created in-and-of-itself (subtracting the satisfaction from making the tool itself)?

dandaka 2 hours ago|

I would love to see a jamming partner. So I could play along him on the same piece.

More comments...