User Tools

Site Tools


concurrent_patch_management

Concurrent Patch Management

For live synth on stage, we have to have the ability to change patches on the fly, in a very reliable and predictable fashion. This page documents one method which is working well as of this writing. It is called “Concurrent Patch Management” because all patches are live and available immediately, even simultaneously given appropriate configuration of MIDI controllers.

History

For quite a few years, I used Robust Session Management to switch between different setups. Its theory is simple: a MIDI/audio toolset is started up automatically at boot, and then when one wants to change a patch, one hits a function key on the keyboard, the existing toolset is shutdown and a new toolset is started up. And it worked quite well; it is still the method I would use if I didn't have the computing horsepower I have now. However, I was noticing after the advent of my current hardware, that I was using less than 1G of RAM, and two or three of the CPUs were not being stressed more than 2-3%. So after my Maximum Strings was well under way, I started to think about how to handle a different problem which, for a long time, had seemed untouchable: fast patch changing. The most important thing which Robust Session Management does not have, is speed; it takes 5-10 seconds to change patches, which is a very long time on the live stage. R.S.M. also had persistent challenges with very big patches, mostly when taking them down: some of the software components sometimes, unpredictably, refused to shut down altogether, and I would have to do a reset to get things working well again. This did not happen very often, but often enough that better was desireable.

Other Approaches, and Seeking a Simpler

Before and during the years in which “Robust Session Management” was in use, there has been much effort in other projects to address the same problem. One of the very promising ones is the Non Session Manager, and quite a few people are using this right now. I tried NSM a number of years ago, and I am sure that today it is working very well for lots of users' needs. There is also the KXStudio tools, especially Claudia and related, which are also in use by many. But all of these are still methods in which we shut down one toolset and start up another, and I ran into so many reliability problems with JACK rewiring and process control during the changes, that I wanted something simpler. And a thought came quite a while ago, given the existence of MIDI channels. Perhaps patch switching could be done at the keyboard, without any changes needed in the synth itself? Might need some major hardware for that…I wonder if Jack Audio runs on IBM mainframes…? :-)

The Theory

It goes like this:

  • The system boots, automatically logs into a GUI desktop.
  • After login, a script named BOOT-INITIAL is run automatically by the desktop manager; BOOT-INITIAL does boot-time configuration and starts up JACK audio; it then calls BOOT-GENERAL to start up the rest of the system. Keeping INITIAL and GENERAL separate aids in debugging.
  • All software elements which will ever be needed in any patch, are always running.
  • All JACK connections, MIDI and audio, which will ever be needed in any patch, are always connected.
  • Each patch consists of one or more chains of software elements.
  • When the musician desires to change the active patch, all he/she does is change the MIDI channel in which his/her keyboard is transmitting. If Patch 1 is desired, the keyboard is to be set to send its signals on MIDI channel 1; Patch 2, channel 2; et cetera up to 16. If we ever get past 16 patches, we'll need something more creative, but this will do nicely for now.
  • Something new, an applet named “Distribute”, routes the MIDI signals to appropriate chains of software elements: data sent by the controller on MIDI channel 1, gets sent to toolchain #1 by the Distribute applet, and ditto for channel 2 / toolchain 2, and so on.

Before implementing the entirety, the first thing to do was to run all software components of all needed patches at the same time, and see if CPU or RAM or anything else got highly stressed. Did that; it appeared OK, just a bit more than half a gig of RAM used, which was peachy, but also used about 28% Jack DSP usage before any sounds were produced! In previous experience I would have considered 28% at zero-tone to be pushing it rather much, but I thought perhaps this box might be different, there were still two or three CPUs not in play very much at all. This turned out to be correct. Also, after a lot more optimization and overall design (more about this shortly), DSP usage after JACK wiring is now about 24%, and it has been singing very well indeed.

Implementation

The toughest part has been making sure all of the elements come up and connect successfully to JACK. If startup is not very carefully implemented, some elements don't come up, or come up and freeze the JACK server, or come up and won't connect to anything, or variations on these themes.

Quite a few different methods have been tested to make the careful startup happen. Originally it was a series of bash scripts just like its precursor Robust Session Management. But it seemed that Python had much more detailed capability to study processes as they ran, and I found that unexpected issues diminished while testing in Python, so BOOT-GENERAL became pure Python.

Methods for controlling and checking processes and monitoring the JACK server, have been written as a special Python module, jpctrl.py (think “JACK Process Control”). Its source has been posted further down; before this I have placed a summary of how things are working.

Here are BOOT-INITIAL and BOOT-GENERAL, discussed immediately.

Run BOOT-INITIAL at system startup

This is done through the desktop manager's start-at-login facility. BOOT-INITIAL resets the log files, kills GVFS if it exists (not in this version 'cause I'm running Sparky/LXDE which makes it easy to remove GVFS!), starts the JACK server using jack_control (which uses some configuration previously saved with QJackCTL), sets several JACK parameters just in case they got changed in development work, starts QJackCTL in DBus mode to permit rewiring and diagnostics, and then runs BOOT-GENERAL.

BOOT-GENERAL is started by BOOT-INITIAL

BOOT-GENERAL contains the real meat of the moment, after JACK itself. Here's a summary of its phases:

First wait for JACK.

wait_for_jack() tries to attach the BOOT-GENERAL process to JACK, once per second for 10 seconds. It returns 0 if successful and 1 if it fails. Then we wait three more seconds just in case, for general stability.

Start Distribute.

Initiate the applet and wait for it to settle, using spawn_and_settle. A number of different methods have been tried to determine the 'settled' status, so far this has worked very much the best. It is important to do this well for each tool loaded, or everything tends to freeze up in a spaghetti-tangle during boot :-)

And then the mixer.

The mixer is a solution to one of the larger challenges of this whole setup, which is mixing the audio streams of the multi-source patches. It is certainly possible to connect the outputs of multiple synths to one port for next steps, either one filter input (e.g. of CalfSRO) or direct to one hardware output. But I have found that use of a good mixer stage to be the most straightforward recipe to prevent overloads of many sorts, starting with simple volume. A good mixer mixes streams together carefully, resulting not in additive volume, but something closely akin to a happy medium, alterable by sliders for each mixer channel.

I tried every single JACK mixer I was able to find over quite a while, and eventually settled on Non-Mixer. Non-Mixer is wonderfully multithreaded; for quite a while in attempting to do its current job I was running one jack-mixer for each primary patch and then routing all of those into a master mixer before sending to the audio hardware.

Patch SRO.

We start all Yoshimi processes for this patch, using spawn_and_settle. Then we start CalfSRO, the filter set.

Next steps.

Then we do the equivalent for patch Strings, including StringsMixer; and also the single Yoshimi for patch FlowBells, which jacks into SROMixer because it shares the CalfSRO filter set.

There are jpctrl.sleep() calls scattered all around just to make sure things come up neatly one after the other. It's possible that some day these might be replaced with a spawn_and_settle with automatic parameters, but reliability is key here, so automation is secondary.

The other element worth mentioning right here is MasterMixer. In the current synth, we have SROMixer with which patch SRO is mixed, and StringsMixer which mixes patch Strings – and both of those have outputs which do not go to hardware, they go to MasterMixer. This is what makes Patch 3, SRO+Strings, work very nicely without overloads, and it will make future patch-work do well as well, with additional mixers to produce mixtures of multiple patches.

And after all of the above are running, and confirmed so as best we can, we run aj-snapshot, which reads file AJBoot.xml, which contains every JACK wire in use, and then sets up all of those wires.

And then it is All Done and Ready to Play!

The Architecture Gives A Bonus: Multipatch Control

If you look at Distribute, you will probably notice that Channel 3 is handled in a special way. Originally, my patch #3 was a customized mix of SRO and Strings; it was customized because SRO and Strings each by themselves are quite beefy, and I had simplified both of them somewhat so to avoid crash&burn at apexes whilst playing Patch 3 :-) But this architecture is very different, and I didn't want to run additional smaller copies of SRO and Strings all of the time.

So I first tried it simply driving the general SRO and general Strings at the same time, from port 3 of Distribute. Sounded good, but overloads abounded. Now the theory behind overloads has always been a problem: there are overloads of simple JACK DSP demand, overloads consisting of too many data streams converging, and others. One of the overload conditions which is very common, though, is simply too much volume: mixing by jack_mixer is set up for each singular patch, but this is a dual patch, two at once, so any pre-existing fixes working fine for either separately, are not going to help with both! So, thought I, how do I reduce the volume deliberately, and in proper proportion, for each of the dual elements?

And I realized that MIDI would do this very nicely, with velocity changes; mididings can handle that too. The above addition within Distribute, is working very well, and is being used for others as well.

Tools In Use

Here are the tools being used:

QJackCTL for setup and diagnostics

QJackCTL is the venerable, often default, and most complete single tool for setup, testing, and configuration of the Jack audio system and ALSA MIDI interprocess/hardware connectivity too. I like QJackCTL for startup and control of the JACK server, but there are elements of today's GUI desktops that will start JACK if it is installed, so we go with the flow and use jack_control.

The Default Install of Jack2

Jack is a background process which connects audio and MIDI applications and hardware. It used to be a simple user-level application, run as needed and then turned off when not. Jack2 is a version rebuilt with multicore CPUs in mind. It is very often started automatically using DBus; its DBus manual control applet is jack_control.

"Distribute", a tool for MIDI distribution.

“Distribute” is a mididings applet, its code being below. After one has installed mididings, one saves the below as a text file, makes it executable, and runs it, and it works very well. Theory items:

  • It has just one MIDI input port, and 16 output ports.
  • If it receives MIDI data on channel 1, it sends that data to output port #1; if it receives data on channel 2, it sends the data to output #2, and so on.
  • To keep configuration of software components as simple as can be managed, it changes all of the data to MIDI channel 1 before outputting to the appropriately chosen port. This also means that all software audio elements are set to the same MIDI channel, 1.
  • There is a special case, where one patch consists of more than one set of software components. You'll see it below; patch #3, for a good example, consists of the sounding elements of patch #1 as well as #2. But if both of those are sounded at full volume, overloads occur rapidly, involving static and potentially crashes! So volumes of both are diminished using reduction of the key velocity values sent by the MIDI controller.
  • Another item which needs special care, is control signals, e.g., the sustain pedal. In the previous version of Distribute, no care was taken with control signals; the sustain pedal changes were sent to whichever output port(s) were active. But this meant that the pedal had to be released, interrupting all tone, before a patch change could be made, or tones would be held by software elements not desired after the patch change. To handle this, a separation was setup in the final area of Distribute where it all comes together, in which all control changes are sent to all active output ports, but all note changes and only note changes, are sent to the patch-appropriate software elements. This is much much nicer, I can change patches on the fly and smoooooothly!

At this writing, patch #6 is not set up yet, but it will combine #1 with a new software element on port 6 to add midrange.

#!/usr/bin/python
 
# Level 1:
# All control changes (e.g., the foot-pedal) are sent to all active JACK ports (currently, 1 2 4 5)
# Everything which is not a control change (e.g., notes) are split:
#	If it comes in on MIDI channel 1, it is designated for applet JACK port 1 (SRO)
#	If it comes in on MIDI channel 2, it is designated for applet JACK port 2 (Strings)
#	If it comes in on MIDI channel 3, it is designated for applet JACK ports 1 and 2 (SRO+Strings)
#	If it comes in on MIDI channel 4, it is designated for applet JACK port 4 (FlowBells)
#	If it comes in on MIDI channel 5, it is designated for applet JACK ports 2 and 4 (Strings+FlowBells)
#	If it comes in on MIDI channel 6, it is designated for applet JACK ports 1 and 5 (SRO+MidOrgan)
 
# Level 2:
# All signals, after leaving Level 1, are designated for MIDI channel 1, and sent out the applet JACK port(s)
# designated in Level 1.
 
# Double patches are described below.  Relative volumes
# are adjusted in MIDI as velocity.  Overload is always a concern.
 
# Channel 3 is SRO+Strings, and uses #1 and #2 in parallel.  
# Trying Port 1 (SRO) at 0.5, Port 2 (Strings) at 0.5.
 
# Channel 5 is Strings+FlowBells, and uses #2 and #4 in parallel.  
# Trying Port 2 (Strings) at 0.85, Port 4 (FlowBells) at 0.35.
 
# Channel 6 is SRO+MidOrgan, and will uses #1 and #6 in parallel.
# No volume changes yet.
 
from mididings import *
 
config(
    backend='jack-rt',
    client_name='Distribute',
    in_ports=1,
    out_ports=16,
)
 
run( 
 
	[
		Filter(CTRL) >> [Port(1), Port(2), Port(4), Port(5)],
		~Filter(CTRL) >> ChannelSplit({
					1: Port(1),
					2: Port(2),
					3: [Port(1) >> Velocity(multiply=0.5), Port(2) >> Velocity(multiply=0.5)],
					4: Port(4),
					5: [Port(2) >> Velocity(multiply=0.55), Port(4)],
					6: [Port(1), Port(6)],
					7: Port(7),
					8: Port(8),
					9: Port(9),
					10: Port(10),
					11: Port(11),
					12: Port(12),
					13: Port(13),
					14: Port(14),
					15: Port(15),
					16: Port(16),
						})
	] >> Channel(1)
 
	)

aj-snapshot for management of Jack and ALSA MIDI connections

aj-snapshot is a command-line and background-service tool which easily saves and loads JACK and MIDI connections. All of the connections for all patches are stored in the one file, AJBoot.xml, and so whenever we make a change – using QJackCTL or other tools, there are a few others – we save our connection set using aj-snapshot:

aj-snapshot AJBoot.xml

BOOT-INITIAL

BOOT-INITIAL is run at system startup, using the start-on-login capability of the desktop manager. GVFS is a function of what we might label fully-functional GUI desktops in X today, purpose being automounting of USB drives and the like, but it can interfere with us, so we kill the processes during this crucial time. In general it is helpful to study the packages you have installed and remove the GVFS backends, this appears to render it harmless.

#!/bin/bash
 
echo "Reset JACK log..." > /home/jeb/boot.log
 
rm /home/jeb/.log/jack/jackdbus.log
touch /home/jeb/.log/jack/jackdbus.log
 
# kill GVFS
/home/jeb/KILLMISC
 
echo "Start JACK..." >> /home/jeb/boot.log
# Start JACK configuration saved using QJackCTL in DBus mode
jack_control start
 
# Start qjackctl
nohup /usr/bin/qjackctl &
 
echo "Starting BOOT-GENERAL..." >> /home/jeb/boot.log
# Start BOOT-GENERAL
/home/jeb/BOOT-GENERAL >> /home/jeb/boot.log

BOOT-GENERAL, the overall startup script

BOOT-GENERAL is run by BOOT-INITIAL, after JACK is started.

#!/usr/bin/env python
 
import os
import jpctrl	# our own Jack Process Control library
 
if jpctrl.wait_for_jack():
	jpctrl.exit_with_beep()
 
print 'Wait 3 seconds for JACK to settle...'
jpctrl.sleep(3)
 
print '-----------------------------------------------------------------'
print 'Start Distribute...'
print '-----------------------------------------------------------------'
 
if jpctrl.spawn_and_settle(
	'/home/jeb/Distribute',
	300, 100, 0, 0): jpctrl.exit_with_beep()
 
if jpctrl.wait_for_jackport('Distribute:out_1') or jpctrl.wait_for_jackport('Distribute:out_16'):
	print 'wait_for_jackport on Distribute failed.'
	jpctrl.exit_with_beep()
else:
	print 'Distribute ports confirmed.'
 
print '-----------------------------------------------------------------'
print 'Start components for patch SRO...'
print '-----------------------------------------------------------------'
 
print 'Start Yoshimi SRO 1...'
if jpctrl.spawn_and_settle(
	'yoshimi -N YoshSRO1 -j -l /home/jeb/YOSHIMI/SROpart1.xmz',
	3400, 400, 0, 0): jpctrl.exit_with_beep()
 
print 'Start Yoshimi SRO 2...'
if jpctrl.spawn_and_settle(
	'yoshimi -N YoshSRO2 -j -l /home/jeb/YOSHIMI/SROpart2.xmz',
	3400, 400, 0, 0): jpctrl.exit_with_beep()
 
print 'Start Yoshimi SRO 3...'
if jpctrl.spawn_and_settle(
	'yoshimi -N YoshSRO3 -j -l /home/jeb/YOSHIMI/SROpart3.xmz',
	3400, 400, 0, 0): jpctrl.exit_with_beep()
 
print 'Start CalfSRO...'
if jpctrl.spawn_and_settle(
	'calfjackhost --client CalfSRO eq12:SRO ! reverb:SRO ! multibandcompressor:SRO',
	700, 200, 0, 0): jpctrl.exit_with_beep()
 
jpctrl.sleep(3)
 
print 'Start SROMixer...'
if jpctrl.spawn_and_settle(
	'jack_mixer -c /home/jeb/SROMixer.jm --no-lash SROMixer',
	900, 400, 0, 0): jpctrl.exit_with_beep()
 
jpctrl.sleep(3)
 
if jpctrl.wait_for_jackport('SROMixer:MAIN R') or jpctrl.wait_for_jackport('SROMixer:MAIN L'):
	print 'wait_for_jackport on SROMixer failed.'
	jpctrl.exit_with_beep()
else:
	print 'SROMixer MAIN ports confirmed.'
 
print '-----------------------------------------------------------------'
print 'Start components for patch Strings...'
print '-----------------------------------------------------------------'
 
print 'Start StringsTreble...'
if jpctrl.spawn_and_settle(
	'calfjackhost --client StringsTreble fluidsynth:1stViolinsStaccato fluidsynth:1stViolinsSustain ' +
	'fluidsynth:2ndViolinsStaccato fluidsynth:2ndViolinsSustain',
	700, 150, 0, 0): jpctrl.exit_with_beep()
 
print 'Start StringsMid...'
if jpctrl.spawn_and_settle(
	'calfjackhost --client StringsMid fluidsynth:ViolasSustain fluidsynth:CelliStaccato	' +
	'fluidsynth:CelliSustain',
	700, 150, 0, 0): jpctrl.exit_with_beep()
 
print 'Start StringsBass...'
if jpctrl.spawn_and_settle(
	'calfjackhost --client StringsBass fluidsynth:BassesStaccato fluidsynth:BassesSustain ' +
	'fluidsynth:BassoonsSustain fluidsynth:ContrabassoonSolo fluidsynth:GeneralBass',
	700, 150, 0, 0): jpctrl.exit_with_beep()
 
# print 'Start Yoshimi for StringsAddBass...'
# if jpctrl.spawn_and_settle(
#	'yoshimi -N StringsAddBass -j -l /home/jeb/YOSHIMI/StringsAddBass.xmz',
#	3400, 400, 0, 0): jpctrl.exit_with_beep()
 
print 'Start MaxStringsFilters...'
if jpctrl.spawn_and_settle(
	'calfjackhost --client MaxStringsFilters eq12:MaxStrings ! reverb:MaxStrings ! multibandcompressor:Strings',
	700, 150, 0, 0): jpctrl.exit_with_beep()
 
jpctrl.sleep(3)
 
print 'Start StringsMixer...'
if jpctrl.spawn_and_settle(
	'jack_mixer -c /home/jeb/StringsMixer.jm --no-lash StringsMixer',
	1300, 500, 0, 0): jpctrl.exit_with_beep()
 
jpctrl.sleep(3)
 
if jpctrl.wait_for_jackport('StringsMixer:MAIN R') or jpctrl.wait_for_jackport('StringsMixer:MAIN L'):
	print 'wait_for_jackport on StringsMixer failed.'
	jpctrl.exit_with_beep()
else:
	print 'StringsMixer MAIN ports confirmed.'
 
print '-----------------------------------------------------------------'
print 'Start component for patch FlowBells...'
print '-----------------------------------------------------------------'
 
print 'Start Yoshimi for FlowBells...'
if jpctrl.spawn_and_settle(
	'yoshimi -N YoshFlowBells -j -l /home/jeb/YOSHIMI/FlowBells.xmz',
	3400, 400, 0, 0): jpctrl.exit_with_beep()
 
print '-----------------------------------------------------------------'
print 'Start MasterMixer...'
print '-----------------------------------------------------------------'
 
jpctrl.sleep(3)
 
print 'Start MasterMixer...'
if jpctrl.spawn_and_settle(
	'jack_mixer -c /home/jeb/MasterMixer.jm --no-lash MasterMixer',
	800, 300, 0, 0): jpctrl.exit_with_beep()
 
jpctrl.sleep(3)
 
if jpctrl.wait_for_jackport('MasterMixer:MAIN R') or jpctrl.wait_for_jackport('MasterMixer:MAIN L'):
	print 'wait_for_jackport on MasterMixer failed.'
	jpctrl.exit_with_beep()
else:
	print 'MasterMixer MAIN ports confirmed.'
 
# Kill GVFS again
jpctrl.spawn_background('/home/jeb/KILLMISC')
 
print '-----------------------------------------------------------------'
print 'And lastly, create JACK connections using aj-snapshot...'
print '-----------------------------------------------------------------'
 
if jpctrl.spawn_background('aj-snapshot -d /home/jeb/AJBoot.xml'):
	jpctrl.exit_with_beep()

jpctrl.py: a JACK Process Control module in Python

Here is where most of the heavy lifting of process control is done. The idea is we have to start each process, gently and invisibly watch it and wait until it settles, and then start the next one. If we don't, some or all of the startup sequence is extremely likely to fail 8-)

#################################
# JACK Process Control module
#
# by Jonathan E. Brickman
#
# version 0.62, 2015-03-09
# (c) 2015 Jonathan E. Brickman
# released under the LGPL:
# https://www.gnu.org/copyleft/lesser.html
#################################
 
from __future__ import division
 
import subprocess
import psutil
import shlex
import jack	# This is PyJack or py-jack, depending on source cited, v0.52
import time
import os
 
# This and the next depend on a Linux command by the name of "beep",
# available in standard Debian repos, not sure about other distros.
def exit_with_beep():
	for count in range(5):
		try:
			os.system('beep')
		except:
			exit(1)
		sleep(0.1)
	exit(1)
 
def beep(freq,length):
	try:
		os.system('beep ' + '-f ' + freq + '-l ' + length)
	except:
		exit(1)
	exit(0)
 
 
# Try to get the JACK port list.  If we're not attached to JACK yet,
# attach to jack, recycle loop and try again.  If jack attachment
# fails, abort.  Return 0 if successful, 1 if fails.
def find_jackport_by_substring(str2find):
	for count in [1, 2]:
		try:
			jportlist = jack.get_ports()
		except jack.NotConnectedError:
			try:
				jack.attach('pyjacksro')
			except:
				print 'wait_for_jackport() could not connect to JACK server'
				print 'aborting.'
				return 1
	for jportname in jportlist:
		if -1 != jportname.find(str2find):
			print jportname
			print 'Flags: ', jack.get_port_flags(jportname)
			print 'IsInput?: ', jack.get_port_flags(jportname) & jack.IsInput
			print 'IsOutput?: ', jack.get_port_flags(jportname) & jack.IsOutput
			print ''
	return 0
 
# A jpctrl-standardized sleep method.
# Accepts fractions as well as positive integers.
def sleep(time_in_secs):
	time.sleep(time_in_secs)
 
 
# Starts a process in the background.
# Return 0 if successful, 1 if fails.
def spawn_background(cmd_and_args):
	if try_popen(cmd_and_args) != None:
		return 0
	else:
		return 1
 
# Start a process in the background, wait until its I/O
# stats can be retrieved, and one more second.
def spawn_and_settle(cmdstr):
	p_popen = try_popen(cmdstr)
	if p_popen == None:
		print 'spawn_and_settle failed for: ', cmdstr
		print 'Could not start process.'
		return 1
	p_psutil = psutil.Process(p_popen.pid)
       	p_io = try_pio(p_psutil)
       	if p_io == None:
       		print 'spawn_and_settle failed for: ', cmdstr
       		print 'Could not get pio data.'
       		return 1
	else:
                print 'spawn_and_settle: process appears ready:', cmdstr
                sleep(1)
		return 0
 
# Tries to get the p_io data used in spawn_and_settle.
# Waits 15 seconds maximum.  This turns out to be valuable
# towards stability, because as load on the system increases,
# processes take longer at startup to become ready
# to give p_io stats.
def try_pio(p_psutil):
	timecount = 0
	while True:
		try:
			print 'Attempting to get pio stats...'
			p_io = p_psutil.io_counters()
		except:
			timecount += 1
			if timecount > 15:
				print 'Could not get pio stats after 15 seconds; aborting.'
				return None
			sleep(1)
			continue
		else:
			print 'successful!'
			return p_io
 
# Tries to start a background process.
def try_popen(cmdstr):
	timecount = 0
        p_popen = subprocess.Popen(shlex.split(cmdstr),-1)
        while True:
        	sleep(1)
        	try:
        		p_psutil = psutil.Process(p_popen.pid)
        	except:
        		timecount += 1
        		if timecount > 5:
        			print 'Could not start ', cmdstr, ' after 5 seconds; aborting.'
        			return None
        		continue
        	else:
        		return p_popen
 
# Waits 6 seconds maximum for the JACK server to become available and
# apparently usable.
def wait_for_jack():
	timecount = 0
	while True:
		try:
			print 'Attempting to attach to JACK server...'
			jack.attach('pyjacksro')
			print 'JACK server discovered, verified, attached!'
			jack.detach()
			return 0
		except:
			timecount += 1
			if timecount > 5:
				print 'JACK server error.  Aborting.'
				return 1
 
# Wait for a particular port to become present in the JACK server.
# Returns 1 on error or 6-second timeout, 0 on success.
def wait_for_jackport(name2chk):
	# Try to get the JACK port list.  If we're not attached to JACK yet,
	# attach to JACK, recycle loop and try again.  If JACK attachment
	# fails, abort
	for count in [1, 2]:
		try:
			jportlist = jack.get_ports()
		except jack.NotConnectedError:
			try:
				jack.attach('pyjacksro')
			except:
				print 'wait_for_jackport() could not connect to JACK server'
				print 'aborting.'
				return 1
	timecount = 0
	while True:
		jportlist = jack.get_ports()
		if timecount > 5: 
			print 'wait_for_jackport timed out waiting for port ', name2chk
			print 'aborting.'
			return 1
		for jportname in jportlist:
			if jportname == name2chk: return 0
		sleep(1)
		timecount += 1

testio: a utility to obtain parameters for spawn_and_settle()

In order to use spawn_and_settle(), we must have one or more numerical definitions of what it means when a process is 'settled': We are doing this for reliable startup, so that each patch element settles before the next one begins loading. It has been established that this is vital for reliability, especially as patches and patch elements get larger and/or more complex. An automatic 'settle' definition based on delta-changes over time was attempted, but was not reliable enough. Here is testio:

#!/usr/bin/env python

import sys
import jpctrl

cmdargcount = len(sys.argv)
if cmdargcount <> 2:
	print "usage: testio '<command_to_run>'"
	print 'note that DOUBLE quotes are required!'

cmdargs = sys.argv
cmdargs.pop(0)

print 'running: ', cmdargs[0]

jpctrl.test_io_counters(cmdargs[0])

So: this is how BOOT-GENERAL was made. The audio element command was run in testio from a shell terminal window thusly:

testio "jack_mixer -c /home/jeb/SROMixer.jm --no-lash SROMixer"

testio will show progressive sets of four numbers, corresponding to the four parameters of spawn_and_settle(). Thus far the first two (read_count and write_count) have appeared the most predictable; although they naturally keep growing after effective 'settling' is done, they grow much slower after a certain big jump, and it is this big jump that we want to represent to spawn_and_settle(). (testio needs a control-C to get it to terminate.) It has been found that the numbers do not have to be very accurate, but each application element has minimum(s) needed to produce overall system stability.

An Overall Summary

Preparation:

  1. We build a good working setup of JACK using QJackCTL, and save it as default. The Primer may help.
  2. We put jpctrl.py and testio into the home directory, and install the py-jack library (also called PyJack), and the beep command for error notification when we eventually run headless.
  3. We run each prospective audio element with testio as documented and use the resultant numbers to build BOOT-GENERAL.

Here's how it flows at run:

  1. Powerup, auto-login into X desktop, auto-run of BOOT-INITIAL.
  2. BOOT-INITIAL starts JACK using jack_control. I found that some distro desktop items are starting the JACK server if it's installed, so we go with that flow.
  3. BOOT-INITIAL then starts BOOT-GENERAL.
  4. BOOT-GENERAL initiates Distribute and then all patch components, carefully waiting until each item is running and settled before starting the next.
  5. BOOT-GENERAL runs aj-snapshot, which wires all audio elements in JACK using settings previously saved in AJBoot.xml.
  6. Music commenceth!!!!!

Summary

In the end, once changes over time are factored in, “Concurrent Patch Management” ends up being much simpler for the system than Robust Session Management. The inter-patch shutdowns and startups of RSM are a much more complicated control problem. The most difficult hurdle for CPM was reliability of startup, because starting everything at once in the background often just did not go, there were frequent freezeups. This was somewhat alleviated by placing one-second delays between loads, but reliability was just not complete. This was especially true with USB MIDI running direct to keyboards, for some reason my Yamaha USB-to-MIDI adapter is more forgiving. But the jpctrl library above, and its ability to identify when processes are 'settled', seems to be just the thing.

The amazing Will Godfrey did suggest that we monitor the stdout of the processes, as a way to determine ready status. This works for Yoshimi, but the other tools in use do not provide such handy feedback. Also, in order to do this reliably in Python I found only the 'pexpect' library to work easily, and unfortunately pexpect-spawned subprocesses are automatically closed on script end.

So until better, here we go, it works it works it works and boy oh boy is it fun!!!!!

concurrent_patch_management.txt · Last modified: 2016/03/08 08:54 by jeb