It’s been fun working on this project! Thanks Josh and Anil for an opportunity to do that! I plan to spend some spare time on it during the term at least…
Main updates and info can be seen in repo (including its READMEs, which I try to keep updated according to the contents).
Having a model that runs on rPi, I decided to test the concept of command recognition.
For simplest setup I’ll use regex to match “start recording”/”stop recording” to control writing to a file.
This can be found in moonshine-enhanced directory, together with README.md. It worked out pretty well).
Another cpp implementation :). Unfortunately, the author of repository developed a actually working example only for Windows, so I’ve spend a some time diving into CMake(a splendid beautiful tool), writing some workaround, and finally rewriting live/file transcriptions.
When testing, I’ve also noticed that the implementation uses only 1 thread while still having RTF=0.56 on 15s audio. Python implementation uses 4 threads, but gives RTF=0.64, which is very strange…
Suprisingly, it took 80s to transcribe a 90s audio. Such inconsistency in RTF introduces a strong need in testing RTF on different audio segments.
Though file transcription works, the live transcription somewhy is inconsistent and should be given a look/reimplemented again.
Live transcription of whisper.cpp (important: should use multithreading effectively, use of all 4 cores didn’t increase performance so far :( ).
Support for audios > 64s for moonshine.cpp at least (other models presumably work ok with that).
Evalution script somewhy produces an ‘object shape’ error when evaluating on the whole dataset. When evaluating on smaller slice - everything evaluates just fine.
RTF evaluation: RTF factor actually depends on the size of a sample, so making a graph would better illustrate the performance patterns.