MSc Interactive Multimedia Susan Knabe 01097503T
Report on final MSc dissertation
Background
Being a musician, the area I want to work in for this dissertation is music synthesis. I am very interested in new technologies concerning the internet and what it allows people to do as well as artificial intelligence and the removal of the need for interfaces between humans and their senses or at least their increased usability and flexibility. My dissertation will combine all of these areas. I am planning to design and build an application that will allow users to draw using different mouse positions, colours, settings and shapes. Whatever they draw will then be translated into algorithms that will synthesise sounds. Creating a specific set of rules and making these known to the user, they will be able to create music just by drawing with a mouse. I want to place this application on the internet so that everybody who is interested in trying it out has access to it. It is my belief that human creativity is limited amongst other things by the need to translate our thoughts and emotions into other media in order to capture them or to communicate them to others. When a musician hears or "feels" music in their mind and wants to capture it they are always faced with the lack of proper means to do so. If they think of a complete piece of music, it is not enough to sing the melody and record it, because it would violate the complexity and the atmospheric properties of the music, neither is it sufficient to write down the musical score, as it would not represent the overall and individual sounds and their attributes. Eventually, by the time they get into a recording studio, looking for the right sounds to represent the product of their mind and having to deal with all the technical issues they might have forgotten what their music felt and sounded like and the piece of work is lost. By providing an increased number of communication platforms between our different "outlets" such as our voice, hands etc, i.e. organs and body parts that can communicate to other people, I believe that we are getting a step closer to breaking down the barriers that limit our power to create art. The ultimate goal would be to produce music directly from our thoughts, e.g. by measuring and translating our brain waves. To some extent, this is possible already. The concept of using BGM (Brain Generated Music) is to set up a device on the user's head that reads their brain waves and computes music accordingly, playing it back to the user in real time. Thus, the user is listening to music that is indirectly produced by their brainwaves. It has been reported that to experience that has an extremely comforting effect and even can significantly decrease blood pressure. Hence, when used regularly, it can have a therapeutic effect. However, if the same user later listens to a playback of their BGM that was recorded during the session, they no longer experience that music the way they did when they were creating it. This effect only occurs due to the music and the brainwaves being in a harmonic rhythm and intensity. However, this does not yet provide the opportunity to directly record music from a person's thoughts, although it might be an advancement into that direction. Why the internet? People spend more and more time on the internet these days, and it seems that the net will be one the main means of communication and of handling daily life quite soon. It is already common to make reservations, purchase tickets, order books and look for the best holidays online. Many even order their food and their shopping over the internet. Young people spend hours online looking for music, games and other activities. Making my application accessible online just seems to be logic in face of all those developments since the nineties. If the prototype were to be simply on a CD, for instance, the number of people who might try it out would be limited to the number of people I would personally give the CD to. If it is online, however, it can be accessed by anybody looking for anything related to it. Furthermore, the application I am planning to make as part of my dissertation is a fairly basic one that can be extended and tailored to suit specific needs. If I had the working prototype online on my own website, I could try to secure some freelance work by contacting other websites that have a music related content. For instance, a popular night club might benefit from having a similar application on their website, allowing clubbers who frequent the venue to draw their own tunes. The interface and the sounds of the music could be adjusted to suit that particular user group by making the application fit into the dance scene, all by simply e-mailing my URL to their website. Technology The application I intend to make aims to build a bridge between our visual perception and our ears by transforming graphical data into sounds. The technology I am going to use for this work is Java. This is due to several reasons: Java is an extremely flexible language that is already extensively and increasingly used both for complex and fully automated back-end computations in business and finance and also multimedia applications like games, animations and sounds. Furthermore, it is compatible across different platforms and systems and it is free. I am hoping to be working on multimedia with Java after this course, so this dissertation is a good preparation and an interesting challenge for that. Although I did not yet have the time necessary to do a lot of research on this topic, I know that Java has the tools needed to synthesise sounds in real time. Areas I will study include the Java Media Framework, JavaSound, JSound and Java 3D, along with other technologies that my research will point me to, such as Pulsar synthesis. One website that holds links to several music applications that have been created using Java is: http://shoko.calarts.edu/~tre/JavaMusic.html Another challenge is making this application run online without the user having to download any plug-ins etc, hence I will try to make all of the prototype's functionalities available as a Java Applet that will execute on the user's computer. Research My research will focus on three main areas: Java technology, synthesis tools, and music applications that are currently available. Looking at the latter will help me make decisions about the functionality of the applet. I need to be clear about many aspects concerning the functionality. For example, will I let users draw something on a static panel and when they want to hear their drawing provide a "Play" button for them to listen back to what they created or will a specified number of seconds loop the music indefinitely with the user simply changing their drawing any time they choose? This decision might affect the complexity of the piece, as I presume that when users use the first option, a static panel, they will put more effort into it and pay more attention to the detail. If the second option is given, they might try out different things more randomly, as the continuous loop might make attention to detail slightly more redundant. Furthermore, how many and which options to change sounds will or can I provide them with? I could make the vertical position (the height along the y-axis on the panel) of the drawn object decide on the pitch, the horizontal position, i.e. how far to the right or the left something is affect the brightness of the sound, or the choice of instrument. The user's choice of colour could affect different filters of sound, for instance, to provide effects on the sounds. Another option is to have a 3D object like a cube in the corner of the panel, that, when moved, can act like a pitch-bend function to the part of the drawing that is currently selected, or it can affect its filters etc. Can I provide a pool of instruments or sounds to choose from? This could be very useful if I were to develop a similar applet as freelance work to clients with particular requirements. I also need to find out how I can let users add beats and rhythm to their piece, possibly by overlaying an optional grid onto the background to enable them to place events in the right position to get a regularly occurring sound. Is it possible to have a built-in option of saving users' work and allowing them access to it when they return to alter it next time they visit the site? I need to study the limitations of using a Java Applet in this way. The other part of studying the functionality is to decide on the design and layout of the interface, e.g. how many options of altering and producing I can include, do I include them as buttons, as 3D objects, as menus etc. The research into the Java technology will inform me about synthesis tools that are available, how they can be used and optimised. For example, using Csound, a music synthesis tool, one has to write and save two different files, a .orc file and a .sco file, both containing code written in a certain syntax. Then, Csound will render those files and save them as a .wav audio file. To open it, the user has to browse to the location of that file and play it back. Csound is therefore not at all suitable for this work, as I need a tool that will synthesise and play back sounds in real time, also allowing for alterations as it is playing back. I need to explore which methods are available and how they do or do not suit my needs to decide which one to use. please e-mail comments to: SusanKnabe@hotmail.com