Every Two Years' LINUX Upgrade

This Blog is about my impressions of an LINUX operating-system upgrade on my Netbook (= small weak laptop) from Lubuntu 14.04 LTS to 16.04. "LTS" means long-term-support, they guarantee you 5 years of minor upgrades. Nevertheless all two years a new major Ubuntu is released, and with it all its variants like Lubuntu (L = "Light").

The LINUX Experiment

LINUX somehow is like an experiment that tries out how many people can work together before the software they produce collapses (like the Tower of Babel did). They are working on a complex and living thing, it's an operating-system. Permanently new hardware has to be integrated and old to be maintained, permanently applications require new capabilities, to be provided by new and (!) old hardware.

Because I believe in this experiment, I always drive my computer by LINUX. It is fast, it is useful, it is free!

Just one thing frightens me periodically, and that's the major version upgrade. With Ubuntu, the currently leading free UNIX, this happens all two years. It was more than once that my LINUX did not boot any more after such an upgrade.

Why Upgrade?

Now you could say "Why being afraid? Just stay on your old version that works perfectly for your hardware!".

No, you can't. Because what you want to do most of all with your computer is use the internet, browse pages containing videos, and be on the always growing communication facilities. Internet applications demand more and more resources, and change technologies quickly and relentlessly.

• Web browsers claim the operating system.
• The operating system claims the hardware to do more and more,
• and this claims us to upgrade.

So an operating system upgrade may not be enough, at some point in time you will even need to buy a new computer, just to be able to use the internet!

Version Information

First thing you want to find out is what operating system version you have. But there is not just one version, there are two versions.

1. LINUX kernel version, e.g. 4.4.0-34-generic
2. Ubuntu version, e.g. 16.04

These versions rarely can be seen together:

• any UNIX kernel version can be displayed by the command-line

  uname -a

• the Ubuntu version can be displayed by clicking system-menu "System Settings" - "Details".
  In Lubuntu, this can be found in system-menu under "System Tools" - "System Performance and Benchmark" - "Operating System".

Online Upgrade

Already having a LINUX on your computer, you can upgrade online over the internet. Just search for the "Software Update" menu item in your system menu. It will automatically search for updates as soon as launched, and present you a list of updates. And it will also report possible kernel upgrades.

Before you make a major version upgrade, be sure that all minor updates are installed. Also make a backup of /boot/grub/grub.cfg, in case you modified this to get a smarter boot menu.

When you started the major upgrade, it runs the same way as the minor upgrade, with the difference that a restart is required after.

Be aware that your computer is now in a critical state, make sure this upgrade is not interrupted, and your internet connection can endure this!

Some Upgrade Observations

The upgrade-tool told me that it's installing the new Ubuntu, not Lubuntu. Which was not nice, because I know that Ubuntu does not work on my Netbook, the hardware is too weak. Finally it turned out that it was NOT an upgrade to Ubuntu ....

---

To watch upgrade-progress, I clicked into the dialog to open the log messages terminal. Strange things appeared here:

....

This likely means that your installation is broken. Try running the command
....

So some upgrade author decided to break the whole installation?!
These messages seem to not mean what they say. A common appearance in software development. It looks like LINUX experts are already used to such wrong alerts. I decided to ignore them, because the terminal messages are not even visible by default.

---

Another thing are these GUI-dialogs suddenly coming up and asking you for "Yes" or "No". You have been going to the kitchen while these downloads proceed so slowly, being back you find it not continued.

So still you need to attend such an upgrade, you can not simply let it run.

---

While watching the terminal log messages I suddenly saw red and blue text on bottom, and no more progress. A closer look showed me that the upgrader again is waiting for my decision, this time not through a GUI-dialog but a terminal-dialog (programmed using the curses-library, the Debian guys do such things). You can not commit such a dialog by mouse. You need to navigate to the OK button using the TAB key, and then press ENTER on it.

So be prepared for all kinds of oddities here. Considering that the log messages terminal not even is visible by default, this is quite demanding!

Summary

After all, my desktop survived the upgrade, and even the boot menu was not again filled with unneeded old version items.
So far the experiment ran quite well :-)

LINUX upgrades always grew stabler over the years, especially since Ubuntu. Hope this will keep on, without LINUX this planet would be a lot darker.

Java TreeMap Comparator Gotcha

You use the Java Comparator interface when you need a certain sort-order in some container. But Comparator does not only declare compare(o1, o2), it implicitly also declares equality of o1 and o2, which is the case when the comparator returns zero (0).

This will not affect you as long as you don't reuse your comparators with different kinds of containers like ArrayList or TreeMap. But when, you may experience the TreeMap gotcha:

• Elements unexpectedly disappear on sorting when using a custom Comparator.

Example Source

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

import java.util.Comparator; import java.util.TreeMap; public class TreeMapComparatorTest { public static void main(String[] args) throws Exception { final String[] toSort = { "start", "end", "employee.name", "job.name", }; final Comparator<String> comparator = new Comparator<String>() { @Override public int compare(String s1, String s2) { return removeAllPrefixes(s1).compareTo(removeAllPrefixes(s2)); } private String removeAllPrefixes(String s) { final int i = s.lastIndexOf("."); return (i >= 0) ? s.substring(i + 1) : s; } }; final TreeMap treeMap = new TreeMap(comparator); for (String s : toSort) treeMap.put(s, "dummy-value"); if (treeMap.size() != toSort.length) throw new IllegalStateException( "Lost some values (" + toSort.length + ") when using TreeMap (" + treeMap.size() + ") !"); } }

What I do here is sort some strings which represent property-names. But I want the "xxx.name" properties to stay together, which would not be the case when using an alphabetical sort order, because then "end" would go between "employee.name" and "job.name". Following is the order I want to achieve (whereby the order of "xxx.name" properties does not matter):

• "end"
• "employee.name"
• "job.name"
• "start"

So I implemented a Comparator that generates such a sort order. It cuts away anything before the last dot, and only then compares the two elements. And it actually works when used with an ArrayList!

But not with TreeMap. When you try out the source code above, you'll see that it throws the IllegalStateException:

Exception in thread "main" java.lang.IllegalStateException: Lost some values (4) when using TreeMap (3) !
 at TreeMapComparatorTest.main(TreeMapComparatorTest.java:32)

When you output the keys of the map, you'll see that one of the "name" properties is missing:

• "end"
• "employee.name"
• "start"

What's happening here?

Responsibilities

I had in mind that comparators are responsible for sorting. That's true, but not the whole truth.

Let's look at the JavaDoc of the compare(T o1, T o2) method of interface Comparator<T>:

Compares its two arguments for order. Returns a negative integer, zero, or a positive integer as the first argument is less than, equal to, or greater than the second.

In the foregoing description, the notation sgn(expression) designates the mathematical signum function, which is defined to return one of -1, 0, or 1 according to whether the value of expression is negative, zero or positive.

The implementor must ensure that sgn(compare(x, y)) == -sgn(compare(y, x)) for all x and y. (This implies that compare(x, y) must throw an exception if and only if compare(y, x) throws an exception.)

The implementor must also ensure that the relation is transitive: ((compare(x, y) > 0) && (compare(y, z) > 0)) implies compare(x, z) > 0.

Finally, the implementor must ensure that compare(x, y) == 0 implies that sgn(compare(x, z)) == sgn(compare(y, z)) for all z.

It is generally the case, but not strictly required that (compare(x, y) == 0) == (x.equals(y)). Generally speaking, any comparator that violates this condition should clearly indicate this fact. The recommended language is "Note: this comparator imposes orderings that are inconsistent with equals."

That's how responsibilities are defined by scientists. Maybe you did as me and stopped reading that complicated text at about 50%. Then you missed the important part that partially explains what happens in the TreeMap gotcha.

It is generally the case, but not strictly required that (compare(x, y) == 0) == (x.equals(y)).

Consequence

What does that mean? You'll find clarity only when you debug TreeMap.put(). Here is an excerpt of its source code:

                int cmp = cpr.compare(key, t.key);
                if (cmp < 0)
                    t = t.left;
                else if (cmp > 0)
                    t = t.right;
                else
                    return t.setValue(value);

You see that this implementation uses the comparator (variable cpr) to detect equal keys. A Map does not allow duplicate keys. Thus it replaces the old value of the key by the new one. And - as a side effect - you lost one element!

As recommended above, you should have added some JavaDoc to your comparator:

"Note: this comparator imposes orderings that are inconsistent with equals."

But would this really have helped :-?

Responsibility Meets Context

You can find a discussion of this gotcha also on stackoverflow. As I said, it will happen when you use Comparator implementations in different contexts:

• in the context of an ArrayList that allows duplicates, the comparator will work,
• but in the a context of TreeMap that allows no duplicates, it might not work.

This "might" is the stuff that unreproducible bugs are made of.

Solution

Following is a workaround:

        final Comparator<String> comparator = new Comparator<String>() {
            @Override
            public int compare(String s1, String s2) {
                final int result = removeAllPrefixes(s1).compareTo(removeAllPrefixes(s2));
                return (result == 0 && s1.equals(s2) == false) ? 1 : result;
            }

            .....
        };

This implementation of compare() avoids to return zero when the given arguments are not equal. Thus "job.name" would not be considered to be equal to "employee.name", and not be removed as duplicate. Nevertheless the sort order is sufficient now:

• "end"
• "employee.name"
• "job.name"
• "start"