Musiah is an expert system (in the AI sense). Expert systems are typically computer programs dedicated to solving problems and giving advice within a specialised area of knowledge with the aim of matching the performance of a human specialist. The field of expert systems is considered by some to be the most advanced part of AI.
The Musiah piano lessons software utilises a number of new approaches to signal and image processing and a number of aspects of artificial intelligence, mainly pattern recognition and inference to overcome a wide range of short-comings in other computer-based tuition applications that have to date prevented artificial tuition of complex subjects such as how to play the piano.
As an A.I. virtual piano teacher, Musiah has the ability to adapt his behaviour to fit new circumstances, and reason, i.e. draw inferences that are relevant to the task or situation at hand. (Giving computers the ability to distinguish the relevant from the irrelevant is one of the hardest problems that has confronted AI to date).
An example of a problem-solving technique used in AI is means-end analysis, which involves the step-by-step reduction of the difference between the current state and the goal state. When assessing a student's playing, using unique algorithms, Musiah compares the current state (what the student has just played) with the goal state (how it should be played), and considers a wide variety of data such as whether the student:
- played additional notes that are not in the sheet music
- omitted some notes
- played some notes incorrectly
- made errors in the timing
- made errors in the duration for which notes are to be held
- paused in the middle of the piece
- played nothing at all
- and every other conceivable variant including whether any or all
of the aforementioned issues occurred in the Left Hand, Right Hand or Both Hands.
Then, depending on the situation, Musiah prioritises which issues to address, in what order to address them, and then proceeds to address them by selecting actions from a list of means - (various ways of guiding/coaching the student) - until the current state is transformed into the goal state.
Musiah overcomes many of the limitations or shortfalls of other currently available products including:
- Limited or no interactivity on currently available products such as "video" lessons
- Inability to monitor what students are playing and provide relevant feedback and guidance during each lesson
- Inability to track student progress after product design is complete to assess product effectiveness in the field and to modify course content as required
- Inability to play music with others as part of the lesson (Musiah includes not only a virtual teacher but also virtual classmates with whom the student interacts. The virtual students also play different parts within ensemble/group pieces along with the student (e.g. in ensemble pieces the virtual students might play bass, strings, brass, woodwind and percussion parts while the student plays a piano part)).
Overcoming these shortfalls in existing products has required extensive experimentation in many areas including but not limited to:
- Enabling interactive communication between MIDI input and a Flash application via a Microsoft.net layer as a translator from MIDI to XML: For Musiah to work, we had to create a new mode of communication between the student's piano keyboard and the embedded Flash application. The area of Flash and MIDI communication is for the most part uncharted territory, with only a few primitive attempts visible within the Flash industry.
- Pattern Recognition/Comparing MIDI files: For the purpose of assessing/scoring a student's performance, a unique system including algorithms for comparing the student's input with a midi format answer file had to be developed.
- Rendering MusicXML in Flash: MusicXML is an emerging industry standard format used to describe/display the appearance of written music. The task of rendering MusicXML in Flash, creating an interactive manuscript was a requirement of the Musiah application so we created an entirely new way of doing this. A Flash MusicXML renderer could also have multiple applications outside of the Musiah application.
- Enabling the program to grow after the software development has been completed: To enable the project to adapt to continually changing student needs and ever evolving lesson scenarios, we have created an admin area with unprecedented functionality to enable authors and curriculum developers (i.e. non-programmers) to input more material for Musiah to teach. Such functionality could have widespread implications outside the project with the potential to dramatically change the way future education/training is undertaken in schools, universities, workplaces and online courses.
To access the program, students will have a musical midi format keyboard connected to their computer. The main application runs on the student's local computer and a secondary application that tracks every student's progress is run via an internet connection. Locally, Musiah will monitor every note played by each student giving them continuous feedback and guidance on how they're going and what they need to do to improve (just like a human piano/keyboard teacher) while also using centrally gathered data to continue to develop and refine the curriculum so the course will continue to improve and evolve.
It is hoped that Musiah will have far reaching applications for other areas of teaching including school education, workplace training, online courses and more.