TL;DR: I tried adding a new KICK state to the solenoid control FSM to mimic the work by Josh Phillips, but I’m not sure whether I’ll keep it. I also tuned the velocity / duty cycle transfer function and ran some more music through the keyboard. This led me to come up with a simple computer vision technique to measure the vertical position of a solenoid plunger over time during a keystroke. I want to repeat this with more data from the control circuit so I can develop a better motion model for higher performance.

🔗Initial KICK State

japhillips87 on GitHub has built a mature and popular version of an electronic player piano. I’ve looked over the design before, but to keep things fun for myself I tried not to look too closely for fear of copying it. Now that my design is showing some viability, I thought it would be alright to at least compare my own MIDI/PWM conversion code with the version Josh built. The big difference is that Josh’s version allows MIDI messages to rewrite solenoid states within a “near future” buffer, which handles some MIDI edge cases at the cost of a small amount of latency. My version has no digital latency but can suffer from latency if note off and note on messages for a given key appear too close together.

Another more minor difference between our designs is that Josh included an extra state between what I call OFF and PRESS. During this KICK state, which only lasts for a few milliseconds, the solenoid is driven at full duty cycle. This is meant to get the key moving quickly and allow greater dynamic range in the MIDI velocity conversion while reducing overall travel time.

I decided to add this state to my own design, but I haven’t committed it yet. I want to run more tests with the measurement setup I describe below to prove to myself whether it’s helpful or not. I think there’s a decent chance that it’s not necessary and the same effect can be achieved by adding some bulk capacitance to the power rail.¹ It would be nice to keep the control FSM as simple as possible and only apply some complexity where it’s really helpful. In any case, the Rust compiler made this change easy by pointing out all the places where the new KeyState::KICK variant wasn’t being handled.

🔗3D-Printed Springs

Some of the keys get stuck down from time to time, and I realized that I could try to mitigate this with some weak compression springs on the solenoid plungers. This is a common feature of solenoids in general, so I gave it a shot with some 3D printer filament. The springs I want are too small for my printer, so instead I tweaked the idea in this video by wrapping two dozen turns or so around a steel rod of the desired diameter and hitting it with a hot air gun until it relaxed into its new shape.^{2 3}

Then I slid the filament off the rod and admired its springiness. This is only a tension spring, though, so I cut the filament into pieces of three turns each and stretched them to length on the solenoid plungers. I didn’t have enough springs for all of the keys, but I decided to try anyway with some longer pieces of music. The system worked fine after I increased the PRESS and HOLD duty cycles slightly to account for the springs acting in opposition to the solenoids.

I don’t think I’m going to use the springs after all. When I was done running the system after about 12 minutes of music (some Brahms this time), some of the solenoids were a little too hot to touch. The solenoids as designed are thermally sensitive to any increase in input power, especially in the HOLD state because it accounts for the overwhelming majority of the time during which the key is active. The springs appear to solve the sticking problem, but they require more force from the solenoids, so I’m probably better off doing more sanding or adding lubrication to reduce the likelihood of sticking in the first place. They do make the keys pop up faster, though, which could be useful for increasing speed.

🔗Vertical Position Measurement

To make meaningful progress towards a better key motion model, I need data. The most important data I need is the vertical position of the key over time while the solenoid is operating. It would also be helpful to have at least some information about the solenoid’s voltage and current. Less important but useful data sources include the PWM input, the commanded control FSM state, and the serial connection to the octave module. Together, these values would provide a good representation of the overall system state and its response to different inputs. I could then use this data to model the system even when the measurements aren’t available, and this would allow the host-side code to produce more faithful outputs given arbitrary MIDI input.

This week I started building a method for at least collecting the vertical position data. I think this is the most difficult data to collect because it’s mechanical and not electrical, the latter of which is relatively easy to collect with a scope or even a microcontroller ADC. Synchronizing the data will be its own challenge, but for now my attention is on the key position data alone. My idea is to capture high-speed video on my phone and use computer vision to extract the plunger’s vertical position from each frame. I can make this computer vision task much easier by attaching a temporary fiducial marker to the plunger extension for the test. My phone’s camera—like most modern devices—can shoot video at 240 frames per second, which gives a time resolution of about 4.2 ms. This is quite good considering the marginal cost, and it gives an acceptable time resolution for analysis.

I (2D) printed an ArUco tag (Garrido-Jurado et. al., 2014), cut it out, and attached it to some cardstock using double-sided tape.⁴ Then, I (3D) printed a basic phone mount and positioned it within about a foot of the key under test. I used a floor lamp to provide some extra light, although this did produce a low-frequency optical beat. The capture procedure was to prepare a test MIDI file to be played on the machine, hit record, play the file, and then stop the recording. Here’s the first viable video I collected, in which the machine plays a key at increasing MIDI velocity. I’ve slowed the video down by 10x. CAUTION: Video contains rapid flashing.

During analysis, I learned that ArUco tags require a “quiet zone” of white area surrounding the black border. I didn’t know that when I made the tag, so my tracking code didn’t work. I was able to salvage the data without manual tracking by using OpenCV’s template matching system. The tag is highly visually distinct by design, so the template matcher had no trouble finding the tag in the scene. In future attempts, I’ll use a proper tag so I can use the ArUco tracker. I might be able to use it to extract pose and scale information more accurately, although I expect that to make only a small improvement to position accuracy.

Using a planar assumption, I converted the vertical pixel positions in each frame to relative positions in real length units. Here’s a selection of plots showing a few interesting parts of that run:

This plot shows the first key press and part of the second. The key starts all the way up, but the solenoid quickly pushes it down and it settles in the down position. Eventually the solenoid releases the key and it begins returning to the up position due to the force of the key action. Before the key can fully return to the up position, the solenoid re-engages and begins driving it downward again. The total travel time for this second key press is about 140 ms, with an impact speed of about 50 mm/s. I think the brief reversal at about 5.5 s is due to a discontinuity in the key action’s force profile. It could also be explained by the transition from the KICK state to the PRESS state, which could conceivably keep the Arduino busy for a handful of milliseconds as it processes the new PWM schedule.

This is the last key strike in the run. The travel time is about 40 ms, with an impact speed around 140 mm/s. This is consistent with the desired output (increasing loudness) and the observed sound. Note that the key reaches a minimum near 20.5 s before suddenly rising by about 0.75 mm around 80 ms later. I think this is due to the fixed duration of the PRESS state (100 ms), which in this case applies substantial force to the key before transitioning to the lighter HOLD state. The key action probably has some small amount of play in it. After the solenoid releases the key for the last time, it is free to bounce slightly before settling into the up position.

I like this plot the best. All points are still equally spaced in time at 240 Hz, but the horizontal axis is now vertical velocity instead of time. This change results in a phase space diagram where a single key press shows up as a counter-clockwise cycle. The cycle begins near the origin, which indicates that the key is in the down position and isn’t moving. When the solenoid releases the key, it begins climbing up the right side of the cycle. I’ve smoothed the velocity slightly, but there are still some interesting oscillations in this region. I don’t know whether these are physically happening or are an artifact of my measurement setup. As the key reaches the up position, the solenoid starts pushing it down again, and the cycle crests to the left. The key picks up a lot of speed on the way down, and the reversal is clearly visible as a sudden shift to the right. The key bottoms out again and comes to rest.

The data I’ve collected so far is interesting and exciting to me, so I want to collect more. This should give me what I need in order to make the host-side code maintain an estimate of the key’s vertical position and velocity over time. That could then feed into an improved key control FSM that at least partly accounts for the key’s mechanics. This all translates (hopefully) into more agile keys, less heat, and better music. I especially like that this can all be handled in software, so it’s relatively easy to change.