When a nerdy dad and 14-year-old music-playing son join forces and start experimenting with music and code, some nice things can happen. Did you ever present your music piece in a business dashboard with charts? Did you know that the FXGL game library can be used to generate a piano with fireworks? And can Virtual Threads play back MIDI events with just a few lines of code and thousands of threads?
About MelodyMatrix
My son wants to become rich and has multiple ideas daily to achieve that. Almost all those ideas are unrealistic, already exist, or very hard to realize. However, occasionally, when such an idea needs a website or some application, I tend to follow his idea and try to create something. In such a case, I use my existing knowledge and add one thing I want to learn.
That’s how we created 4DRUMS, a website to share drum videos. I used Spring Boot, Vaadin, and PostgreSQL to build the complete system during a few evenings and I learned how to use the APIs of YouTube and Vimeo so we didn’t need to render and host the video files ourselves. It’s a nice project, but as it goes with such websites, the programming was the easy part. Attracting users and making money out of it will probably never happen…
My son also loves to create YouTube movies with piano music, so he came up with another idea: “Can we create an application to visualize music in different ways?” As a JavaFX lover, this immediately triggered me, and I started experimenting with MIDI and different ways to visualize the data that can be received from a musical instrument. That’s how MelodyMatrix was born! Using the same tools used for 4DRUMS, I created the website melodymatrix.rocks, where you can find more information about the app and where you can download it.
The website’s “Thanks to…” page lists all the libraries and tools used for the application and website.
About MIDI
The Musical Instrument Digital Interface (MIDI) standard has existed for a long time and defines how musical instruments and controllers (PCs) can interact with each other and share data. The main thing you should know is the data format. With each press or release of a key on, e.g., a piano, a message is sent with three bytes of data:
- Status (4 bits) + Channel (4 bits)
- Data 1
- Data 2
The data values are used differently depending on the status. A good article with much more detail can be found on songstuff.com.
OpenJDK includes code to interact with MIDI devices and handle MIDI files. You can find the sources in the OpenJDK GitHub repository, where you can see that this code is quite old. The stability of the MIDI standard and its implementation in Java result in the fact that there haven’t been any changes in this part of OpenJDK. A more modern implementation of MIDI on the JMV is provided by ktmidi by Atsushi Eno, a Kotlin Multiplatform library for MIDI 1.0 and MIDI 2.0.
In Foojay podcast #54: Music and MIDI with Java and Kotlin, I talked with Atsushi and Geert Bevin about using MIDI with Java. Geert is a Belgian Java Champion who moved to the US and now works for Moog Music, which creates synthesizers and other musical instruments. You may also know him as the creator of RIFE2 and bld, two other amazing Java projects.
Building Blocks
I once learned that focusing on one thing at a time is vital to mastering something new in a project. So I decided to build MelodyMatrix with Java, JavaFX, with Kotlin as “the new thing”. This is inspired by the fact that the ktmidi library is also written in Kotlin, and it can be mixed with the many existing and wonderful Java libraries created by the community, like the Charts library by Gerrit Grunwald and the FXGL game library by Almas Baim. The complete list of libraries used in the app and website is listed here.
The desktop app installer is created with jDeploy and runs on GitHub Actions to fully automate the distribution of new versions and update the already installed ones.
While looking for a way to sell licenses for the app, I learned about “Merchants of Record”. This is a company that handles the sales of digital products, makes invoices, takes care of taxes, etc. I use Polar, which claims to be “the fastest way to add SaaS & digital products to your stack”. They are a young company, share much of what they do as open-source, and are very responsive if you need help. So, although we haven’t sold licenses yet, I’m pleased with their service!
More Readable Code With Kotlin
The most important advantage of using Kotlin in this project is the .apply {} approach that leads to more readable code. When using JavaFX, you end up with a lot of code that initializes a UI component and then applies a set of options. The following code generates the same UI but is written more cleanly.
The code in Java:
var borderPane = new BorderPane();
var buttons = new VBox();
buttons.setSpacing(10);
buttons.getChildren().addAll(
new Button("Button 1"),
new Button("Button 2"),
new Button("Button 3")
);
borderPane.setLeft(buttons);
The same in Kotlin, using .apply {} to remove the repetition of the object names:
var borderPane = BorderPane().apply {
left = VBox().apply {
spacing = 10.0
children.addAll(
Button("Button 1"),
Button("Button 2"),
Button("Button 3")
)
}
}
In my opinion, the second code is cleaner. I shared a more extended example in a blog post and video some months ago but got some mixed reactions… 😉
Playing Music With Virtual Threads
As MelodyMatrix is a new project, I started with the latest Java Long Term Support (LTS) version: 21. This unexpectedly helped me solve a coding challenge very easily! As explained before, MIDI events only contain three bytes of data. In MelodyMatrix, you can make a recording, which stores these events as a record with the timestamp of the event and the data. Represented as JSON, it looks like this:
{
"name": "Test recording",
"start": 1711203466078511000,
"data": [
{
"t": 1711203466078534000,
"d": [-112, 60, 29]
},
{
"t": 1711203466480248000,
"d": [-112, 60, 0]
},
...
]
}
So, a recording can be visualized as a timeline of data packets:
I needed to find a way to play back a recording, send it to an instrument, and generate the visual effects. I considered an approach with some continuous loop to go through the list of events, send them as MIDI data as soon as the timestamp has passed, and mark them as handled so they would no longer be evaluated in the loop. But a quick experiment with virtual threads revealed that a much simpler approach can be used!
What Are Virtual Threads?
Virtual Threads were introduced as a preview feature in OpenJDK 19 as part of the Project Loom. They became fully integrated in OpenJDK 21. These virtual threads are also called “Lightweight Threads” as they are constructed as Java objects and take far fewer resources than traditional threads. The JVM runtime manages them, and they have no one-to-one mapping with the OS threads. As soon as their task blocks while waiting for an API response, database query, file to open,… the JVM will put them back into a “todo list”, and handle other tasks.
They are ideal for concurrency use cases where you must switch between many tasks. But you should not use them for long-running tasks that will never block, as the virtual thread system will cause overhead in such cases.
Virtual Threads in MelodyMatrix
When you start playing a recording in MelodyMatrix, a virtual thread is constructed for all the events. So, each press or release of a music key or pedal becomes a task to be handled.
// A list to store all the threads,
// so we can interrupt them if we want to stop the playback.
var playThreads: MutableList<Thread> = mutableListOf()
// Thread factory with a custom naming pattern.
// The trailing number will increment starting from 0.
val factory = Thread.ofVirtual().name("recording-player-", 0).factory()
// Executor service to create a new thread for each submitted task.
val executor = Executors.newThreadPerTaskExecutor(factory)
// Timestamp of the first data point in the recording,
// used to calculate relative timings for the playback tasks.
val recordingStart = recording.data.first().timestamp
// Create a task for each event with the time difference
// between the first event and the current one.
recording.data.forEach { d ->
val task = MidiDataPlayer(d, d.timestamp - recordingStart)
val thread = Thread.startVirtualThread(task)
executor.submit(thread)
playThreads.add(thread)
}
What happens in each of these tasks is basically… sleeping! 😉 The task waits till the time has passed that it has to wait to send the MIDI data to the instrument. This is a blocking action, so a perfect use-case for Virtual Threads and the JVM can easily handle thousands of these events and handle them at the right moment.
override fun run() {
try {
Thread.sleep(seconds, nanos.toInt())
midiHandler.play(dataLine, delayInNanos)
} catch (e: InterruptedException) {
// Nothing, just accept the interruption
}
}
Is this the perfect implementation for playing back music? Maybe not, and there are probably much smarter approaches to achieve this. But it works! And as a good team leader once said, “If it works, don’t touch it…”
Conclusion
Java and JavaFX are excellent combinations for creating a user interface application. By adding Kotlin, I could learn something new and create more readable and maintainable code. The application is far from finished and will probably never get finished, as most pet projects…. But it’s an ideal way to learn how to turn an idea into a sellable project and all the side activities involved, like marketing, creating videos, talking about it at conferences, etc.
If you are into music, please download the free version. Give it a try, and give us some feedback! Check this GitHub repository to see how the views are created. Maybe you can even make a pull request to improve them or add a new one?
If you want to learn more about MelodyMatrix or see it in action, take a look at this recording of the Devoxx talk we gave in Antwerp, Belgium, in October:
steinhauer.software
