M4L module for Ableton Live which enables basic gesture trackingand mapping. The module is based on the gesture-follower patch ofthe IRCAM's FTM library and it makes use of the native capabilitiesof the Live API. The mapping between the gestures and audibleoutput is designed within the module and it enables the simultaneouscontrol of macro features of the Ableton interface such as clipselection, volume and panning of the track/s.

Test it with an iPad or iPhone app that can send OSC messages

(Step 1) download, extract and move to Cycling74 folder [THIS].

(Step 2) download [M4L module] and place it into ~/Library/ApplicationSupport/Ableton/Library/Presets/MIDIEffects.

(Step 3) you can use it immediately with these TouchOSC patches:[iPhone]

Download [PDF] of below paper submitted to the 50th AISB Symposium.

Introduction

Machine Learning (ML) toolkits such as Wekinator [11], SARC EyeswebCatalog[3], IRCAM MnM toolbox[4] and OpenCV [8] are betweenthe most well-known toolkits available to artists and engineers[2]. Most of these tools have Open Sound Control (OSC) [9] capabilitieswhich allow them to communicate with third-party softwarewhich create and manage the audio/visual elements of the performance.The process connecting gestures to audio-visual outputs isgenerally dealt with ad hoc algorithms to suits the performance orthe performer's needs.The module described in this paper, ML-AL, is based on the IRCAM'sMnM toolbox and it exploits the M4L [6] capabilities to directlyinterface gesture recognition algorithms with the audio/visualelements within the software Ableton Live [5]. In that regard themodule does not present a new algorithm or a novel approach in MachineLearning but rather implements existing technologies withinone popular music software. However, the design of the module offersan approach to the use of gesture tracking algorithms for the discretecontrol of system's parameters rather than a continuous soundgenerating mechanism and/or continuos controllers.

ML-AL FEATURES The ML-AL module has three sections each of which is dedicated toa specific task. These are visually separated by the blue vertical linesas depicted in Fig.1. The sections are: 1. Data-Input: it serves the purpose of retrieving data from a connecteddevice via OSC.2. Gesture Recognition: it performs gesture recognition analysis ona user-built gesture dictionary.3. Mapping: it links the result of the performed gesture analysis to aseries of Ableton Live native functions.

Data-Input The data input section receives from any device that can transmitaccording to the OSC protocol. In this section the user can decide through which port number the communication needs to be established.The data needs to be formatted according to the followingOSC address:  1. /naccxyz: this is the data which will be read by the gesture recognitionsoftware for the creation and subsequent anaysis of the gestures.It needs to be a list of three numbers (e.g. the accelerationreadings from a 3-axis accelerometer).  2. /n1n/ push2: The sender device must have a push button whichworks as a gate. This push button serves the purpose of clearlymarking beginning and end points of a gesture by letting the datathrough when depressed and stopping the data when released.

Gesture Recognition The gesture recognition section is based on the gesture-follower examplepatch offered in the IRCAM's MnM toolkit. This toolkit waschosen because fully compatible with the M4L package available inAbleton Live. The patch proved to be a reliable and a ready-madesoftware that could successfully implement gesture recognition. Furthermore,it is written in Max and this allows for easy manipulationand addition of user elements. It is important to notice that the patchis not making explicit use of the time progression features that it enables[1]. Rather, the patch is used as a gesture classifier.The author acknowledges the limitations that such a system imposes.This choice was dictated by privileging artistic needs overtechnical possibilities. In particular, machine learning was thought, atleast in this early version of the module, to be useful for the discretecontrol of macro elements of a live performance, such as the triggeringof pre-recorded loops, rather then acting upon an eventual soundgenerating mechanisms or continuous controllers. This is a strategythat the authors has found productive in severals other works [10].

MappingThe data used for the control of the Ableton interface is the likeliestnumber value retrieved by the gesture analysis routine performed bythe gesture recognition section and made available for mapping eachtime the push button is released.The ML-AL module can store up to eighty presets. These presetscan store the start/stop mode of a clip, volume and panning of up toeight different audio or MIDI Ableton tracks. The likeliest numbervalue is then used to recall one of these presets on-the-fly.MODE OF USE The presented module has been tested and interfaced with an iPadrunning a custom patch made with TouchOSC and complying withthe specifications outlined in Section 2.1. The mode of operation of the module consists of the followingsteps:1. load the module onto a MIDI track.2. input OSC port number over which to establish the connection(data-input section)3. enable 'Learn' mode in the gesture recognition section4. enable 'start' (toogle) and perform a gestures.5. disable 'start' (toggle)6. repeat steps 3 and 4 for as many gestures as you require. Makesure that to each gesture corresponds a different phrase number.7. disable 'Learn' modeBefore using the module in performance mode, it is required tostore some presets (preferably of equal quantity of the gestures performedduring the training steps).1. create up to eight audio or midi tracks in Ableton Live.2. import or create as many clip as desired in each track3. create a combination of playing clips, volume and panning settingsand store these to a preset number (shift+click on one of the circlesin the ML-AL module preset object).4. repeat step 3Now the module is ready to be used in performance mode. Enablestart in the Gesture Recognition section and play a gesture. Thegesture number performed, if successfully recognised, will recall theequivalent preset number.

Performance Scenario The ML-AL module was preliminary tested by developing a shortalgorithmic process controlled by gestures performed using an iPad..Four audio tracks were added to the Ableton project and each filledwith two clips with distinct sound properties (high drones, lowdrones , granular, irregular percussive patterns). The following configurationwas thought as 4-bit resolution system (four tracks by twoclips and with a clip per track always playing) given therefore a totalof sixteen possible combinations of playing clips stored as presets.The ML-AL module was trained with sixteen different gestureseach of which mapped to a preset number. At performance time, themodule response was fast and with a good rate of accuracy in recognisinggestures. A drawback for the system is represented by the effortrequired by the user to learn sixteen different gestures. However,the quality of a live performance is not necessarily measured by thequantity of gestures performable or performed. The module capabilities,in conjunction with more traditional continuos controllers (faders) for the manipulation of audio effects, offers already a goodsound palette for performance purposes.CONCLUSION This paper has presented and described the features of a new M4Lmodule for Ableton Live which enables basic gesture recognition andmapping. The module, based on the IRCAM's MnM toolkit, offerssimple and easy to use mapping of recognised gestures to basic controlparameters available in Ableton Live such as clip selection, volumeand panning settings for up to eight tracks. The module makesuse of a highly efficient gesture recognition algorithm working on anon-continuos classification method. The mapping algorithm enablesquick recall of Ableton presets so to, for example, control the macroelements of a performance such as the intervention of scenes and/or aselected combination of clips. The perceived latency during preliminarytesting was found neglegible thus making the module suitablefor live performances. ML-AL is freely available at [7].

REFERENCES [1] F. Bevilacqua, B. Zamborlin, A. Sypniewski, N. Schnell, F. Guedy, andN. Rasamimanana, 'Continuous realtime gesture following and recognition',in Gesture in Embodied Communication and Human-ComputerInteraction, eds., Stefan Kopp and Ipke Wachsmuth, volume 5934 ofLecture Notes in Computer Science, 73–84, Springer Berlin Heidelberg,(2010).[2] B. Caramiaux and A. Tanaka, 'Machine Learning of Musical Gestures',in Proceeding of New Interface for Musical Expression Conference,NIME'13, (May 2013 KAIST, Daejeon, Korea,).[3] Eyesweb Catalog. available at: http://www.somasa.qub.ac.uk/ngillian/sec.html [accessed 27th of January 2014].[4] Ircam Ftm. available at: http://ftm.ircam.fr/ [accessed 27th of January2014].[5] Ableton Live. available at: https://www.ableton.com/ [accessed 27th ofJanuary 2014].[6] Max for Live. available at: https://www.ableton.com/en/live/maxforlive/[accessed 27th of January 2014].[7] ML-AL. available at: http://muresearchlab.com/softwares/mlal/ [accessed27th of January 2014].[8] OpenCV. available at: http://opencv.org/ [accessed 27th of January2014].[9] OpenSoundControl. available at: http://opensoundcontrol.org/ [accessed27th of January 2014].[10] G. Torre, The Design of a New Musical Glove: A Live PerformanceApproach, (Ph.D. Thesis) University of Limerick, 2013.[11] The Wekinator. available at: http://wekinator.cs.princeton.edu/ [accessed27th of January 2014].