Performing an update on a rolling release GNU/Linux distribution never gets boring. And while using such a distribution in production is a clear no-go in most cases, doing this on a development machine ensures that our debugging skills stay sharp. There is also this minor inconvenience of not being able to do our work while things are broken, but totally worth it ;)
What did we break this time
Like most developers who occasionally work remotely, we rely on a VPN to be able to access the juicy parts of XLAB’s infrastructure. It is no secret that we are using openfortivpn via NetworkManager and we even described some of the issues we came across when we started using it.
Usually, we were greeted by this password input dialog when we tried to establish the VPN connection:
But not this time. This time we were presented with this “Photoshopped” dialog:
Oh yeah, let the games begin ;)
First things first
As with every error that we encounter, we first searched the internet for poor
souls that might have been in the same situation as we are right now and
steal copy their solution. Unfortunately, the search turned up nothing.
To give search engines something to chew on and index, we filled a bug report and packed it with as much information as we had at the moment.
Filling the bug report is probably the most important thing to do when we encounter the problem, since this will leave a trail that other users/developers can follow. Remember: If it is not in the bug report, it did not happen!
With the bug being properly reported, we focused our efforts on solving the issue.
Getting our password input back
User interface parts of the NetworkManager are written using GTK+ library, which is great news. Why? Because GTK+ ships with an integrated inspector that we can use to mess around with applications that are running.
There was just one small thing that we needed to do before we could start inspecting the misbehaving dialog: activate the inspector keybinding. Fortunately, this was as simple as running
$ gsettings set org.gtk.Settings.Debug enable-inspector-keybinding true
Now we were able to click our way to the dialog and start inspecting its
internals by pressing
Ctrl+Shift+D. And bingo: just after a few clicks, we
discovered that the password input was still present in the dialog, but hidden
for some reason. Manually flipping the
visible property to
TRUE made the
input visible again and we were able to establish a connection.
Now we knew what was wrong, but we still had no clue why things broke.
Determining the root cause
We knew that things were working just fine until the latest update. Thanks to
emerge logs (yes, we use this rolling distribution ;) and our
habit of doing regular updates we were able to reduce the number of suspects
down to a single package: nm-applet.
To confirm our suspicions, we downgraded the
nm-applet package, and sure
enough, things magically started working again.
So the next thing we did was to clone the network-manager-applet
repository and search for string
password_entry that we discovered while
inspecting the dialog. Somewhere in the middle of the
git grep "\bpassword_entry\b" output we found this beauty:
.../nma-vpn-password-dialog.ui: <object class="GtkEntry" id="password_entry">
For those who are new to GTK+,
*.ui files contain graphical interface
definitions and are usually created using Glade. And since something went
wrong with the UI of our dialog, this file looked like a good candidate to
To test our hypothesis that the UI definition change broke our dialog, we ran
$ git log -p src/libnma/nma-vpn-password-dialog.ui
and quickly found this in commit 07d900c0:
@@ -113,9 +112,8 @@
<object class="GtkEntry" id="password_entry">
- <property name="visible">True</property>
+ <property name="visible">False</property>
Great, now we knew what (probably) broke our dialog (and for the
non-programmers who stuck with us this far: changing value of the
False is bad). All that remained now was to fix the
Preparing a fix (or workaround)
Since change was made in the
network-manager-applet, one would think that
the fix should also be placed there. But upon further inspection, the commit
that caused the password entry to remain hidden was sensible and removed a
bunch of deprecated stuff, which is always a good thing.
So instead of changing the applet code, we decided to fix the VPN dialog
itself. We cloned the fortisslvpn repo and started looking for that broken
dialog. Luckily for us, there exists an
auth-dialog folder right in the root
of the project, containing a single file.
After a bit of grepping in the
network-manager-applet repository, we managed
to produce this simple fix for our dialog troubles. But how to test it?
Luckily, we are running Gentoo, so compiling the fixed version of the VPN plugin required no additional packages to be installed on the system and was as simple as running:
$ ./autogen.sh && make
After the compilation finished, we ended up with a bunch of files that were
useless to us, with one exception:
This was the patched dialog executable that should fix our problem. In order
to test it, we copied it to the
(overwriting the original executable without making a backup copy, because why
not ;) and tried to establish the connection again.
And it worked like a charm. We created a new merge request in the fortisslvpn repository, pat ourselves on the back and started working on things that we were supposed to be working on for the last two hours ;)
What did we learn from this endeavor
If were were to emphasise a single lesson learned while solving this bug, it would definitely be:
Bundling debug tools inside production-ready packages is awesome and we should start doing it too.
Being able to inspect the dialog using GTK+ inspector significantly reduced the time needed to find a workaround and determine what exactly broke. So thanks again GTK+ developers for making our lives a bit easier.