Season of KDE 2024 Blog #1: Introduction and current progress

Who am I?

I’m aisuneko icecat (that alias is actually a reference to GNU IceCat), a Chinese first year university student, amateur developer and passionate user / supporter of open source software for years.

I’ve switched to a FOSS-only workflow since 11th grade, with Arch Linux, LineageOS Android and KDE Plasma as my daily driver. I’m also on my way to learning more diverse and modern skills and tech stack such as Python and Rust. My hobbies range from computing, software development to graphic design (which made me a frequent Inkscape user), electronic music, rail transit, anime and more.

Feel free to check out my GitHub or reach out to me via Telegram and Email :)

What am I working on for this year’s SoK?

My project is about adding support for a keyframe curve editor and enhancing existing support of advanced keyframe types in Kdenlive, the popular open source video editor in the KDE suite which I’ve also been using for quite some time.

Kdenlive’s keyframing mechanism allows certain parameters of an effect in a video to change over time - the position and geometry of a clip, for example, could be modified over time by defining keyframes to create an animation. Until recently, Kdenlive offered only three relatively basic types of interpolation between two keyframes - linear, discrete and smooth. Luckily, recent MLT (Kdenlive’s backend framework) releases added support for some diverse easing functions, rendering it possible to create more advanced animations in Kdenlive. A feature still not implemented yet frequently requested by the community is a bézier curve editor allowing full customization of the interpolation method of a keyframe, which is what I’ll be working on in this project.

I’ve maintained fundamental knowledge in C/C++ through my days doing competitive programming, and have got my hands wet on the Qt framework through several hobby projects I did in high school, which in turn motivated me to sign-up for Season of KDE. Not only do I hope to help bring the said feature into reality, but also to sharpen my own development skills through real-world projects, contribute back to the open source community and gain more inner confidence.

My mentors for the project are Julius Künzel and Jean-Baptiste Mardelle, the two leading developers of Kdenlive. They are both very friendly and helpful to work with, and I appreciate my opportunity of collaborating with and learning from them in the process.

My work done so far

Setting up the development environment

It’s fairly easy to set up the development environment by following the Kdenlive build guide on GitLab, except a few notable caveats:

• It is not necessary to use Qt 6 in this project and Qt 5.15 should work just fine. To disable Qt6 and force Qt5 add the -DQT_MAJ OR_VERSION=5 -DBUILD_WITH_QT6=OFF flags to CMake args.
• It is advised to use the ninja build system instead of make, for the former only does incremental compilations instead of recompiling everything from scratch on every change, drastically speeding up compile time on every build. To do this use the -GNinja flag in CMake command to generate build files for ninja, then type ninja to build. Compiling tests also take quite some time and could be disabled with -DBUILD_TESTING=OFF.
• On non-Ubuntu distributions, certain packages e.g. qt6-5compat not documented in the build guide might be missing - even under a fully working KDE Plasma install - and will result in CMake errors.

My main development tool is Qt Creator, a powerful and handy C/C++/Qt IDE with lots of batteries included, making it the perfect choice for developing KDE applications. It comes with a built-in documentation viewer but will sometimes freeze on my machine, so I occasionally use Qt Assistant (the standalone version of the said viewer) for reading Qt docs.

Reading through existing codebase

A major portion of my time spent on the project goes to reading through existing codebase of Kdenlive and it’s backend MLT. It takes some effort to understand the keyframing architecture of Kdenlive and the structure of its corresponding GUI, since there is very little documentation of the codebase itself. Here are some notes I took in the progress.

MLT keyframing fundamentals

MLT’s animation and keyframing system is defined in mlt_animation.c and mlt_animation.h, with the help of MLT’s extensible data handling framework mlt_properties.

There are three types of interpolable data defined in mlt_properties.h: rect (x, y, width, height, opacity), color (r, g, b, alpha), numeric (just a value). When processing keyframe data and interpolating related animation objects, MLT follows a top-down model of “mlt_animation -> animation_node (a linked list) -> mlt_animation_item (holds its type and data)” - data is gradually de-encapsulated as it percolates down each layer, and the data of neighboring keyframes are also used in calculation.

The raw MLT keyframe data, retrievable via Kdenlive’s “Copy keyframes to clipboard” function for example, resembles something like this:

"0=-658 175 1920 1080 0.000000;82B=584 5 1920 1080 1.000000;301q=226 -323 1920 1080 1.000000;691g=-291 -202 1920 1080 0.619048;916t=-271 440 1920 1080 0.000000"

Different keyframes are separated with semicolon. The general data format in each one is as follows:

• the current frame number (relative to the beginning of the effect)
• a character denoting the mlt_keyframe_type (defined in this enum; omitted on linear keyframes)
• all items following the = symbol are the mlt_property data of the keyframe; eg. the above example shows a rect property

Keyframing architecture in Kdenlive

Kdenlive offers a myriad of effects and transitions for the user to adjust video clips to their liking. Once applied on a clip, their settings are accessible through a side panel (typically at the bottom right of Kdenlive’s main interface). Each of the “settings widget” in the said panel are instances of AssetParameterView and within each one resides an AbstractParamWidget, the base class of all widgets representing a parameter of effects, transitions etc.

A keyframable effect is controlled with a KeyframeWidget (defined in src/assets/view/widgets/keyframewidget.cpp) which apparently inherits from AbstractParamWidget. KeyframeWidget houses and maintains connections with a KeyframeView, the existing timeline-style keyframe controller, which is actually a canvas painted by a QPainter. This is also where my new widget will reside.

The actual keyframe data it manipulates is managed by the lower-level interface KeyframeModelList (which includes several KeyframeModels), accessed by higher-level GUI through shared pointers. Multiple keyframable parameters in an effect have different QPersistentModelIndexs, allowing us to create individual keyframe curve editors for each of those parameters. All those components are closely connected with each other through Qt’s signals-and-slots mechanism so that user’s changes could be applied to (and shown on) clips in real time.

Implementing changes

During the above process, I decided to start with small changes to familiarize myself with the general workflow. I noticed that the developers used a quick but dirty way to add QAction handles of MLT keyframe types in Kdenlive, which isn’t quite extensible. As a result, I refactored related code and added a tooltip in GUI as a proof of concept, then submitted my changes as my first merge request in the project.

After that, I was away for two weeks before starting to work on the actual widget. I decided to base my widget on existing code - that’s right, there are already some code for building sufficient “bézier curve editor” UI in Kdenlive’s codebase, primarily for controlling the frei0r effect of the same name.

It is based on a slightly different architecture compared to the one mentioned above. The said UI is named BezierSplineEditor (defined in src/assets/view/widgets/curves/bezier/beziersplineeditor.cpp) which uses a custom data structure called CubicBezierSpline to keep track of the current curve data. BezierSplineEditor is inherited from AbstractCurveWidget which spawns a generic “curve editor” widget. It could optionally be wrapped in a CurveParamWidget which provides additional controls (e.g. value spinner boxes, tool buttons etc) beneath the canvas, though not necessary in this case. In short, I will use code from BezierSplineEditor to lay the foundation (e.g. canvas initialization) for my widget and implement rest of the functions myself.

We need to properly initialize and position the new widget (called KeyframeCurveEditor for now, and is currently a stub) in KeyframeWidget first. I created a QStackedWidget in the place of KeyframeView as a container of both the latter and my widget. I defined a button and a function called KeyframeWidget::slotToggleView to enable switching between the two types of editor views.

Here’s when I encountered a major hurdle: There might be several instances of KeyframeCurveEditor, one for each “subparameter” (x, y, width, etc), within a single KeyframeWidget. After hours of digging through existing code, I’m still not able to understand how those “subparameters” are added (and separated), and was stuck for a few days. With the help of my mentor, I finally figured out how these individual parameters are added to our KeyframeWidget, and managed to create my own widget (actually a QTabWidget container for them) inside the KeyframeWidget::addParameter function.

After the widget is initialized, it’s time to load our keyframe data into the widget for display. To do this I wrote the helper function loadSplineFromModel which generates the widget’s main CubicBezierSpline given existing data from KeyframeModel, and managed to have it rendered on the widget. I also made various UI enhancements, such as adding two text labels to indicate value boundaries.

So far I’ve completed most of the UI implementation for KeyframeCurveEditor. It looks like this:

(There are notable gaps among those interpolated (grey) points on the left; I suspect it’s an issue with the way of loading data from KeyframeModel. Not sure about it though)

What’s next?

In the remaining three weeks of SoK, I plan to connect the widget to the rest of the keyframing system and do neccessary changes to render it actually usable. This will be the more intriguing and perplexing part, and I’m making some slow progress already ;)

“But what about the part with MLT?” That’s right, I still need to make relevant changes to the MLT framework (this will also require me sending a PR to the MLT developers) in order to have a fully customizable “bézier curve” keyframe type, but I haven’t came up with an optimal solution for that… yet. In light of my current progress, I’m afraid that I might need to save it for a separate project after SoK… Though either way, I’m looking forward to continue working with the Kdenlive team to further polish the feature before releasing it to end users.

The project turned out to be quite challenging for me. I’ve had quite some headaches reading through Kdenlive’s codebase and thinking of ways to implement my planned features. I’m still trying my best to tackle with various difficulties in the progress. Regardless of the outcome, I’m glad to be offered the opportunity to work with such an open source project through Season of KDE, and this will surely serve as valuable experience for me on my path forward.