Controlling Sonic Pi from Scratch 3 on a Raspberry Pi4

IMPORTANT
Note this project is designed for Scratch 3 running on a Raspberry Pi computer. Because it will also be running Sonic PI, a 2Gb or 4Gb RAM machine is preferable, although I have used the program on a 1Gb Pi3B+ as well. At the time of writing (May 2020) The Sonic PI version 3.1 supplied with Raspbian will NOT work. Instead you should download and install Sonic Pi 3.2.2 from my site here. If you are unable to or don’t want to do this, you can run Sonic Pi 3.2.2 on a Mac or PC on a different computer on the same local network and use the option to start the linking python server with appropriate IP addresses.

Several years ago I had a brief play with an OSC interface which let you communicate with an early version of Scratch. Unfortunately it was not compatible with later versions of Scratch, but Scratch3 on the Raspberry Pi now has several add-ons incorporated including one to allow it to interact with the GPIO pins. I have already done several projects with Sonic Pi together with a python script acting as an OSC server and client to allow Sonic PI to communicate with the GPIO pins using the gpoizero library so I decided to experiment and see if I could produce something which would let it communicate with Sonic Pi. The result is this project, which lets Scratch3 act as a virtual keyboard for Sonic Pi, which can either be used to play notes, or as a trigger to control the sounds that Sonic Pi is producing. It is not perfect and suffers from a certain amount of latency, which is unfortunately variable, but nevertheless is fun to do, and I think that the end result will be appealing to scratch and sonic-pi users alike.

The project consists of three parts. First some programming in scratch, much of which is repetitive, with the same code being associated with each of the 20 “keys” on the keyboard and some further code on the “main stage”. Secondly, the python script, which uses the python-osc library to support the OSC communications, and the gpiozero library to interact with the GPIO pins. Thirdly some code running in a buffer in Sonic PI. This has essentially two OSC messages it can send to the python script which are used to start and stop the interaction, and it also responds to two different incoming OSC messages which indicate the state of a particular key and whether it is on or off. Beyond that the code depends on what you wnat Sonic Pi to do in response to these inputs, whether to play as a straight keyboard instrument, or whether to choose from a range of musical responses when it detects inputs.

Looking at the Scratch3 code first. Here are pictures of the stage with its two “costumes”

As you can see there are 20 green buttons, each of which represents a different note in teh range. The main note keys are named, with the interspersed smaller buttons representing #/flat notes. When the mouse is held over one of these buttons with the mouse button down, it alters one of the GPIO pins and the python script sends an appropriate message to Sonic PI. When the button is released (or the pointer is moved off the green button, the GPIO pin reverts to its normal state, and another message is sent to Sonic Pi.

The next picture shows the code associated with the stage.If you click it, you can see that this has four elements in it. Top left is a forever loop that continuously monitors GPIO pin 26 (bcp naming). If this is low it switches the stage costume to the red background image, otherwise it switches to the blue background image. The state of the GPIO pin is governed by incoming OSC messages from Sonic Pi, via the python script. Below that is some code which receives a broadcast message from the reset X button. When this is pressed and the broadcast message is received it changes GPIO pin from hight to low for 0.4 seconds, before returning it to a high state. This is used to send an OSC message  “/reset” from Scratch 3 to Sonic PI via the Python script. The two pieces of code top right continuously reset a timer while the Green Flag is active, and if the timer has ran for 0.1 seconds without a reset then switches the stage costume back to blue.The reset button code is shown above (click to enlarge) , sending the Broadcast message when it is clicked.

The heart of the code is associated with each of the 20 green buttons. You can click it to see an enlarged copy. It shows the code for button1. Every button has the same myblock defined, named blippin (short for blip input). Looking at this block code, it has one parameter which is named pin. This is the number of the GPIO pin (in BCP numbering) associated with the particular green button in question. For button1 it is in fact GPIO pin BCP 1. The my block is called by the associated code consisting of a Green Flag followed by blippin 1
The first action line of the blippin block in light blue moves the green sprite to an x y position which is stored in two lists one of x the other of y coordinates, with an associated lookup list which relates the GPIO pin number (which follows a sequence I chose) of 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20 for the pins in use. I chose these to avoid pins used by the Pimoroni Fan Shim which I have connected to my Pi4 so that they could work in harmony together. Using the lookup list which contains:
1,2,0,3,4,5,6,7,8,9,10,11,12,0,0,13,0,0,14,15,16,17,18,19,20 lets the correct entry in the x,y coordinates to be chosen. So for example if we are looking at the sprite Button3 this actually has 4 for its pin number. The list shows entry 4 is the number 3, so it will chose the third entry in the x,y coordinate lists. The reason for having these lists and the goto x,y command is that it is very easy to inadvertently move the buttons, and this ensures that their positions are reset each time the program is run. The beginnings of the three lists are shown below.again, you can click the image to see a larger version. Once the position of the button has been adjusted, the next line of the blippin myblock starts a forever loop which continuously controls the state of the GPIO pin associated with the block. If the mouse pointer is inside the green sprite and the mouse button is down, it changes the state of the pin to be low, and keeps it there until either the mouse button is released or the pointer is moved off the green sprite.

code. That completes the Scratch3. However, before loading the file you should select the Raspberry Pi GPIO extension from the blue extensions tab bottom left on the Scratch3 screen, as this code is essential for it to work.

Turning now to the python script, the code for this is shown below:

#!/usr/bin/env python3
#SPoschandler.py written by Robin Newman, May 2020 
#Provides the "glue" to enable the GPIO on Raspberry Pi
#to communicate with Sonic Pi. Sonic Pi can control LEDs etc,and receive
#input from devices like push buttons connected to GPIO pins
#Sonic Pi can be running either on the Raspberry Pi,
#or on an external networked computer

#The program requires gpiozero (already in Raspbian) and python-osc to be installed


from gpiozero import ButtonBoard,LED,Button
from pythonosc import osc_message_builder
from pythonosc import udp_client
from pythonosc import dispatcher
from pythonosc import osc_server
from time import sleep
import argparse
import sys


bn=20 #number of "buttons" in ButtonBoard
b=ButtonBoard(1,2,4,5,6,7,8,9,10,11,12,13,16,19,20,21,22,23,24,25)
reset = Button(27) #dealt with separately from ButtonBoard
flag=False #used in def pr to separate on/off messages
activate=False #used to activate sender once it is declared in __init__
current=0 #used to hold current activated button

def pr():
  global flag,current
  if activate: #make sure sender has been defined and is active
    if flag==False:
      for i in range(0,bn): #find which pin triggered on
        if b.value[i]>0:
            print(i,b.value[i])
            sender.send_message('/playOn',i) #send OSC message for on
            flag=True #switch to looking for "off"
            current=i #current 'on' pin
    else:
      flag=False #looking for switch off
      print(current,b.value[current])
      sender.send_message('/playOff',current)
    
b.when_pressed = pr #trigger change of state event handled by function pr

def doReset(): #deals with reset button pushed (pin 27)
    global activate 
    print("reset",reset.value)
    if activate==True: #only proceed if sender is activated
        sender.send_message('/reset',1) #send reset OSC to Sonic PI


reset.when_pressed=doReset #triggers doReset when button is pressed

l1 = LED(26) #this pin state controlled by input OSC from Sonic Pi


   
 #This is activated when /start OSC message is received by the server.
 #single argument is 1 or 0
def start(unused_addr,args, n):
    print("Start",n)
    if n==1:
        print("l1 on")
        l1.on() #set GPIO pin 26
    if n==0:
        print("l1 off")
        l1.off() #reset GPIO pin 26
        
#The main routine called when the program starts up follows
if __name__ == "__main__":
    try: #use try...except to handle possible errors
        #first set up and deal with input args when program starts
        parser = argparse.ArgumentParser()
        #This arg gets the server IP address to use. 127.0.0.1 or
        #The local IP address of the PI, required when using external Sonic Pi
        parser.add_argument("--ip",
        default="127.0.0.1", help="The ip to listen on")
        #This is the port on which the server listens. Usually 8000 is OK
        #but you can specify a different one
        parser.add_argument("--port",
              type=int, default=8000, help="The port to listen on")
        #This is the IP address of the machine running Sonic Pi if remote
        #or you can omit if using Sonic Pi on the local Pi.
        parser.add_argument("--sp",
              default="127.0.0.1", help="The ip Sonic Pi is on")
        args = parser.parse_args()
        if args.ip=="127.0.0.1" and args.sp !="127.0.0.1":
            #You must specify the local IP address of the Pi if trying to use
            #the program with a remote Sonic Pi aon an external computer
            raise AttributeError("--ip arg must specify actual local machine ip if using remote SP, not 127.0.0.1")
        #Provide feed back to the user on the setup being used    
        if args.sp == "127.0.0.1":
            spip=args.ip
            print("local machine used for SP",spip)  
        else:
            spip=args.sp
            print("remote_host for SP is",args.sp)
        #setup a sender udp-client to send out OSC messages to Sonic Pi
        #Sonic Pi listens on port 4560 for incoming OSC messages
        sender=udp_client.SimpleUDPClient(spip,4560) #sender set up for specified IP
        activate=True #signal that sender is now defined, to functions above
 
        #dispatcher reacts to incoming OSC messages and then allocates
        #different handler routines to deal with them
        dispatcher = dispatcher.Dispatcher()
        #The following handler responds to the OSC message /testprint
        #and prints it plus any arguments (data) sent with the message
        dispatcher.map("/testprint",print)
 
        #following dispatcher handles "/start" osc message
        dispatcher.map("/start",start,"n")

        #Now set up and run the OSC server
        server = osc_server.ThreadingOSCUDPServer(
              (args.ip, args.port), dispatcher)
        print("Serving on {}".format(server.server_address))
        #run the server "forever" (till stopped by pressing ctrl-C)
        server.serve_forever()
    #deal with some error events
    except KeyboardInterrupt:
        print("\nServer stopped") #stop program with ctrl+C
    #Used the AttributeError to specify problems with the local ip address
    except AttributeError as err:
        print(err.args[0])
    #handle errors generated by the server
    except OSError as err:
       print("OSC server error",err.args)
    #anything else falls through

The OSC server is set up in the __main__ part of the script towards the end. This first of all sets up a parser to receive and analyse the input parameters. Often none will be supplied if Scratch2 and Sonic PI (3.2.2) are running on the same Pi and default values will be employed (127.0.0.1 or local host for the ip address on which they are both running, port 4560 for osc messages sent to Sonic PI and port 8000 for messages sent to the python server). However, if these are on different machines, then you must specify the ip address of the Pi and the ip address of the computer running Sonic Pi using syntax as below.

For both on local machine you can use

./osc-ScratchandSonicPi.py

and on different machines

./osc-ScratchandSonicPi --ip 192.168.1.150 --sp 192.168.1.129

IN this latter case 192.160.1.150 would be the ip address of the Pi and 192.168.1.129 the ip address of the computer running Sonic Pi on the same local network. Note the execute bit has to be set on the script to run it directly like this

chmod +x osc-ScratchandSonicPi.py

or you can use

python3 osc-ScratchandSonicPi.py

instead. After the arguments have been read and parsed, the Sonic Pi ip address is used to initialise a  sender in the line

sender=udp_client.SimpleUDPClient(spip,4560)

This sets a udp client to send OSC messages back to Sonic Pi on its IP address held in the spip variable and to port 4560 on which Sonic PI is listening for incoming messages. Once this set up, a flag activate is set to true, signifying to functions in the first part of the script that the sender is initialised.
Next two dispatcher routine are set up. These are called in response to a matching OSC message being received from Sonic Pi. One is a test message addressed to “/testprint” any number or test string following this will merely be printed on the terminal window running the OSC server, as the name suggests just to test whether it is working. The other incoming OSC message the server will respond to is “/start” This is followed by a single numeric parameter  n, which should either be 0 or 1 to adjust the state of GPIO input pin 26 (l1). This will be passed onto a function send defined in the top half of the program.
The main function in the top half of the script is def pr() This is triggered whenever the state of any of the GPIO pins defined as a gpiozero ButtonBoard is altered. The routine determines which button has been altered and sends an OSC message “/playOn” or “/playOff” to Sonic Pi with the index of the pin in the ButtonBoard list as a parameter. The system depends on the fact that only one of the green sprites at a time is activated, so having received an “on” signal from one pin it switches to waiting for an “off” signal from the same pin. The other two functions in the first part of the script deal with detecting that the reset button in the Scratch program has been pressed def doReset() which sends the OSC message ‘/reset’ to Sonic PI with the parameter 1, and the function def start which sets the state of GPIO pin 26 according to the incoming OSC message “/start” from Sonic PI discussed earlier when discussing the dispatchers.

Turning to the Sonic PI program, the content of this depends upon what you want it to do when it interacts with the Scratch keyboard. The specimen script supplied has two alternatives. The first reads the data received by the “/playOn” and “/playOff” incoming OSC messages and uses them to control a note played by the :dsaw synth.

#Sonic Pi Scratch3 interface by Robin Newman, May 2020
#two programs. Uncomment one section OR the other at a time
#first section sets Sonic PI as a keyboard instrument with dsaw synth
#second section has selectors to play sample groups, chords and two sample sub programs
use_osc "localhost",8000
#osc "/testprint","hello there"
osc "/start",0
set :flag,true
define :groove1 do
  live_loop :g1 do
    synth :tb303,note: scale(:c3,:minor_pentatonic,num_octaves: 3).choose,release: 0.2,cutoff: rrand_i(60,110)
    sleep 0.2
    stop if get(:kill)
  end
end


define :groove2 do
  l1=(ring 1,0,1,0,1,0,1,0,1,0,1,0)
  l2=(ring 0,1,0,1,0,1,0,1,0,1,0,1)
  l3=(ring 0,1,0,0,1,1,0,1,0,1,0,1)
  l4=(ring 1,0,1,0,1,1,0,1,0,1,0,1)
  l=(ring l1,l2,l3,l4)
  
  live_loop :drums1 do
    r=l.tick(:l)
    24.times do
      stop if get(:kill2) #check for when to stop this thread
      tick
      a=0.5;a=1 if look%3==0
      sample :drum_tom_hi_hard,amp: a,pan: [-1,1].choose  if r.look==1
      sleep 0.1
    end
  end
  
  live_loop :drums2 do
    stop if get(:kill2) #check for when to stop this thread
    a=0.5;a=1 if tick%4==0
    sample :drum_tom_lo_hard,amp: a,pan: [-0.5,0.5].choose
    sleep 0.3
  end
end

live_loop :waitReset do
  use_real_time
  r = sync "/osc*/reset"
  set :flag,false
  osc "/start",1
end

with_fx :reverb,room: 0.8,mix: 0.7 do
  uncomment do #uncomment for dsaw synth player
    use_synth :dsaw
    
    live_loop :test do
      use_real_time
      k = sync "/osc*/playOn"
      if get(:flag) == true
        puts k[0]
        z= play 60+k[0],sustain: 5
        k=sync "/osc*/playOff"
        control z,amp: 0,amp_slide: 0.05
        sleep 0.05
        kill z
      else
        stop
      end
    end
  end
  
  comment do #uncomment for sample /chords/live_loop player
    live_loop :test2 do
      use_real_time
      k= sync "/osc*/playOn"
      if get(:flag) == true
        p=k[0]
        puts p
        case p
        when 0
          z=sample (sample_names sample_groups[0]).choose
        when 1
          z=sample (sample_names sample_groups[1]).choose
        when 2
          z=sample (sample_names sample_groups[2]).choose
        when 3
          z=sample (sample_names sample_groups[3]).choose
        when 4
          z=sample (sample_names sample_groups[4]).choose
        when 5
          z=sample (sample_names sample_groups[5]).choose
        when 6
          z=sample (sample_names sample_groups[6]).choose
        when 7
          z=sample (sample_names sample_groups[7]).choose
        when 8
          z=sample (sample_names sample_groups[8]).choose
        when 9
          z=sample (sample_names sample_groups[9]).choose
        when 10
          z=sample (sample_names sample_groups[10]).choose
          
        when 11
          z=sample (sample_names sample_groups[11]).choose
        when 12
          z=sample (sample_names sample_groups[12]).choose
        when 13
          z=sample (sample_names sample_groups[13]).choose
        when 14
          z = synth :tb303,note: chord_degree([1,3,5,8].choose,:c3,:major,3),sustain: 5,cutoff: rrand_i(80,100)
        when 15
          z = synth :fm,note: chord_degree([1,3,5,8].choose,:c4,:major,3),sustain: 5,cutoff: rrand_i(80,100)
        when 16
          z = synth :zawa,note: chord_degree([1,3,5,8].choose,:c3,:major,3),sustain: 5,cutoff: rrand_i(80,100)
        when 17
          z = synth :mod_saw,note: chord_degree([1,3,5,8].choose,:c3,:major,3),sustain: 5,cutoff: rrand_i(80,100)
        when 18
          set :kill,false #flag used for killing groove1
          groove1
        when 19
          set :kill2,false #flag ued for killinggroove2
          groove2
        else
          puts p
        end
        k=sync "/osc*/playOff"
        if k[0]<18 #keys that handle stamples
          control z,amp: 0,amp_slide: 0.05
          sleep 0.05
          kill z #kill smple after quick fade
        end
        if k[0]==18 #keys that start live loop functions
          set :kill,true #kill groove1
        end
        if k[0]==19 #keys that start live loop functions
          set :kill2,true #kill groove2
        end
      end
    end
  end
end

The program starts by defining the address of the Python server (here “localhost” and its associated port 8000. An optional osc message to /testprint follows to check the python server. osc “/start”,0 sets the stage of the Scratch3 program to red, and a :flag time-state is set to true to enable the Sonic PI program.
Two functions groove1 and groove2 are defined which are used in the second alternative of the  program, more of which later.
The main part of the program starts with the fx :reverb line which adds some reverb to all that is played. Initially the first of the two alternate programs is uncommented, and the synth :dsaw is selected. The live_loop :test follows, This is set to use real time to give the fastest response. It waits from an incoming osc message “/playOn” which is detected in Sonic Pi by the line

k = sync "/osc*/playOn"

When this detects a message the incoming parameter held in the list k is retrieved using k[0] and is used to start a note playing with pitch 60+k[0]. Since the parameter received is the index position of the sprite which has been clicked this will produce a note whose pitch depends on the numerical position of the sprite in the list. The note is set to have a duration of 5 seconds, and a reference to the playing note is stored in the variable z. The live loop then awaits the arrival of an OSC message addressed to “/playOff” detected by the line

k=sync "/osc*/playOff"

When this is received the note first has its volume (:amp) reduced and is then stopped by the code

control z,amp: 0,amp_slide: 0.05
sleep 0.05
kill z

If the :flag referred to above is not true then the program is inhibited and the live_loop stopped. So this program will continuously respond to input from the scratch3 program playing notes as directed.
The alternative program which can be manually uncommented (whilst commenting the first part) use the keys as selectors to control the selection of a ruby case statement used in live_loop :test. Again it starts by setting use real time and checking for the state of the :flag. It takes the value of the incoming “/playOn” OSC message parameter to make the selection. The first 14 selections each choose one of the sample_groups in Sonic Pi.:ambi, :bass, :bd, :drum, :elec, :glitch, :guit, :loop, :mehackit, :misc, :perc, :sn, :tabla, :vinyl
For the selected sample group it then chooses at random ONE of the relevant samples and plays it. It keeps a reference z to the playing sample and stops it playing if an incoming “\playOff” message is received before the sample finishes.
For inputs from sprites with indices 14 to 17 it plays a chord_degree in c major, selecting degree 1,3,5 or 8 at random. Each of these uses a different synth. In the case of the :fm synth the degree is up an octave as that synth normally plays one octave down.
The last two sprites indices 18 and 19 each trigger a function either groove1 or groove2. Each of these start an enclosed live_loop or loops playing. When either of these keys is released then set a time state variable :kill to true. This is used stop the relevant live_loops from playing. Groove1 has a simple live_loop that plays a sequence of notes chosen at random from a c minor_pentatonic scale, whereas groove2 plays two more complex live_loops producing patterns of tomtoms playing.
The final loop to remark on is named :waitReset, and this one waits for an incoming osc message “/reset” which sets the :flag to false, thus inhibiting the Sonic Pi program, and getting it to send back an osc message “/start”,1 which sets the scratch stage to blue signifying control has stopped.
Note: all received osc messages in Sonic PI have a section prepended to the address. Thus “/playOn” might be received as /osc:192.168.1.150:36419/playOff [0] Since we don’t need the initial information about ip address and port we detect it with the aid of a wild card * using “/osc*/playOn”

If you want to run Sonic PI on a different computer on the same network as the raspberry Pi then change line the initial line use_osc “localhost”,8000 to use the ip address of the Pi running Scratch3 eg use_osc “192.168.1.150”,8000 In this case start the python script with parameters indicating both the ip address of the Pi and the address of the Sonic Pi computer.eg ./oscScratchandSP.py –ip 192.168.1.150 –sp 192.168.1.129 (in the example 192.168.1.150 is the address of the Pi and 192.168.1.129 is the address of the Sonic Pi computer).

All the code is available from my github site
A video of the project in action is available here
Final note. Since writing the article and making the video I have slightly altered the position of the sprites on the screen. This affects the values in the ylist in the scratch program but nothing else.