Removing Unused LINUX Kernels by Shell Script

When you list your LINUX kernels you will find out that upgrading to a new version does not remove the old kernel. It stays on disk and inside the grub boot-manager menu as a fallback-system. This may become a problem when your /boot file-system is of restricted size (maybe due to a dedicated boot-partition). Anyway these kernels are not needed in case your LINUX starts up nicely after upgrade.

This Blog is about how to remove unused kernels.
Mind that the preceding '$' in all commands below is the UNIX terminal prompt and does not belong to the command itself.

Listing Kernels

I did this on Ubuntu, which is a Debian system, thus I used dpkg to list kernels.

$ dpkg --list | grep "linux-" | grep "image-"
rc  linux-image-3.13.0-32-generic              3.13.0-32.57                                 amd64        Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc  linux-image-3.13.0-43-generic              3.13.0-43.72                                 amd64        Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc  linux-image-3.13.0-44-generic              3.13.0-44.73                                 amd64        Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc  linux-image-3.13.0-45-generic              3.13.0-45.74                                 amd64        Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc  linux-image-3.13.0-46-generic              3.13.0-46.79                                 amd64        Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc  linux-image-3.13.0-53-generic              3.13.0-53.89                                 amd64        Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc  linux-image-3.13.0-74-generic              3.13.0-74.118                                amd64        Linux kernel image for version 3.13.0 on 64 bit x86 SMP
ii  linux-image-3.13.0-92-generic              3.13.0-92.139                                amd64        Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc  linux-image-4.15.0-42-generic              4.15.0-42.45                                 amd64        Signed kernel image generic
ii  linux-image-4.15.0-43-generic              4.15.0-43.46                                 amd64        Signed kernel image generic
ii  linux-image-4.15.0-48-generic              4.15.0-48.51                                 amd64        Signed kernel image generic
rc  linux-image-4.4.0-124-generic              4.4.0-124.148                                amd64        Linux kernel image for version 4.4.0 on 64 bit x86 SMP
rc  linux-image-4.4.0-128-generic              4.4.0-128.154                                amd64        Linux kernel image for version 4.4.0 on 64 bit x86 SMP
rc  linux-image-4.4.0-131-generic              4.4.0-131.157                                amd64        Linux kernel image for version 4.4.0 on 64 bit x86 SMP
rc  linux-image-4.4.0-138-generic              4.4.0-138.164                                amd64        Linux kernel image for version 4.4.0 on 64 bit x86 SMP
rc  linux-image-4.4.0-140-generic              4.4.0-140.166                                amd64        Linux kernel image for version 4.4.0 on 64 bit x86 SMP
rc  linux-image-4.4.0-31-generic               4.4.0-31.50                                  amd64        Linux kernel image for version 4.4.0 on 64 bit x86 SMP
rc  linux-image-4.4.0-34-generic               4.4.0-34.53                                  amd64        Linux kernel image for version 4.4.0 on 64 bit x86 SMP
rc  linux-image-4.4.0-83-generic               4.4.0-83.106                                 amd64        Linux kernel image for version 4.4.0 on 64 bit x86 SMP
rc  linux-image-4.4.0-87-generic               4.4.0-87.110                                 amd64        Linux kernel image for version 4.4.0 on 64 bit x86 SMP
rc  linux-image-4.4.0-92-generic               4.4.0-92.115                                 amd64        Linux kernel image for version 4.4.0 on 64 bit x86 SMP
rc  linux-image-4.4.0-97-generic               4.4.0-97.120                                 amd64        Linux kernel image for version 4.4.0 on 64 bit x86 SMP
rc  linux-image-4.4.0-98-generic               4.4.0-98.121                                 amd64        Linux kernel image for version 4.4.0 on 64 bit x86 SMP
rc  linux-image-extra-3.13.0-32-generic        3.13.0-32.57                                 amd64        Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc  linux-image-extra-3.13.0-43-generic        3.13.0-43.72                                 amd64        Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc  linux-image-extra-3.13.0-44-generic        3.13.0-44.73                                 amd64        Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc  linux-image-extra-3.13.0-45-generic        3.13.0-45.74                                 amd64        Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc  linux-image-extra-3.13.0-46-generic        3.13.0-46.79                                 amd64        Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc  linux-image-extra-3.13.0-53-generic        3.13.0-53.89                                 amd64        Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc  linux-image-extra-3.13.0-74-generic        3.13.0-74.118                                amd64        Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
ii  linux-image-extra-3.13.0-92-generic        3.13.0-92.139                                amd64        Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc  linux-image-extra-4.4.0-124-generic        4.4.0-124.148                                amd64        Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP
rc  linux-image-extra-4.4.0-128-generic        4.4.0-128.154                                amd64        Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP
rc  linux-image-extra-4.4.0-131-generic        4.4.0-131.157                                amd64        Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP
rc  linux-image-extra-4.4.0-138-generic        4.4.0-138.164                                amd64        Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP
rc  linux-image-extra-4.4.0-140-generic        4.4.0-140.166                                amd64        Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP
rc  linux-image-extra-4.4.0-31-generic         4.4.0-31.50                                  amd64        Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP
rc  linux-image-extra-4.4.0-34-generic         4.4.0-34.53                                  amd64        Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP
rc  linux-image-extra-4.4.0-83-generic         4.4.0-83.106                                 amd64        Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP
rc  linux-image-extra-4.4.0-87-generic         4.4.0-87.110                                 amd64        Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP
rc  linux-image-extra-4.4.0-92-generic         4.4.0-92.115                                 amd64        Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP
rc  linux-image-extra-4.4.0-97-generic         4.4.0-97.120                                 amd64        Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP
rc  linux-image-extra-4.4.0-98-generic         4.4.0-98.121                                 amd64        Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP
ii  linux-image-generic                        4.15.0.48.50                                 amd64        Generic Linux kernel image
ii  linux-signed-generic                       4.15.0.48.50                                 amd64        Complete Signed Generic Linux kernel and headers (dummy transitional package)
rc  linux-signed-image-3.13.0-44-generic       3.13.0-44.73                                 amd64        Signed kernel image generic
rc  linux-signed-image-3.13.0-45-generic       3.13.0-45.74                                 amd64        Signed kernel image generic
rc  linux-signed-image-3.13.0-46-generic       3.13.0-46.79                                 amd64        Signed kernel image generic
rc  linux-signed-image-3.13.0-53-generic       3.13.0-53.89                                 amd64        Signed kernel image generic
rc  linux-signed-image-3.13.0-74-generic       3.13.0-74.118                                amd64        Signed kernel image generic
rc  linux-signed-image-4.4.0-124-generic       4.4.0-124.148                                amd64        Signed kernel image generic
rc  linux-signed-image-4.4.0-128-generic       4.4.0-128.154                                amd64        Signed kernel image generic
rc  linux-signed-image-4.4.0-131-generic       4.4.0-131.157                                amd64        Signed kernel image generic
rc  linux-signed-image-4.4.0-138-generic       4.4.0-138.164                                amd64        Signed kernel image generic
rc  linux-signed-image-4.4.0-140-generic       4.4.0-140.166                                amd64        Signed kernel image generic
rc  linux-signed-image-4.4.0-31-generic        4.4.0-31.50                                  amd64        Signed kernel image generic
rc  linux-signed-image-4.4.0-34-generic        4.4.0-34.53                                  amd64        Signed kernel image generic
rc  linux-signed-image-4.4.0-83-generic        4.4.0-83.106                                 amd64        Signed kernel image generic
rc  linux-signed-image-4.4.0-87-generic        4.4.0-87.110                                 amd64        Signed kernel image generic
rc  linux-signed-image-4.4.0-92-generic        4.4.0-92.115                                 amd64        Signed kernel image generic
rc  linux-signed-image-4.4.0-97-generic        4.4.0-97.120                                 amd64        Signed kernel image generic
rc  linux-signed-image-4.4.0-98-generic        4.4.0-98.121                                 amd64        Signed kernel image generic

When you find yourself with such a big list of kernels you don't want to remove them one by one. But before you apply a script you should try regular commands.

Regular Clean-Up Commands

First try this command:

$ sudo apt autoremove

Now check if they are gone. Listing /boot shows existing kernel files:

$ ls -la /boot

Any file in that directory that is named vmlinuz is a kernel. It should carry the version in its name. It is accompanied by other files like initrd.img, so you should always remove a kernel by system-commands, not by a file-remove.

If old kernels are still present, try this command:

$ sudo apt-get autoremove --purge

Still not gone? Now you would need to remove them one by one, e.g. for linux-image-3.13.0-32-generic:

$ sudo apt-get -y --purge remove linux-image-3.13.0-32-generic

But in case you have as many old kernels as I had, you may appreciate following script.

Clean-Up by Script

Don't ever remove the current kernel! You can find out its version by this command:

$ uname -r
4.15.0-48-generic

Also don't remove linux-image-generic.

To avoid any misconception, the following script just outputs command lines. Redirecting and executing the result is left up to you.

purge-generate.sh

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

[ -z "$1" ] && { echo "SYNTAX: purge-generate.sh kernelversionhead" >&2 echo " Example: purge-generate.sh 4.4.0" >&2 exit 1 } KERNELVERSION=$1 list=`dpkg --list | grep linux- | grep image-` kernels=`echo "\$list" | awk ' /linux-image-'\$KERNELVERSION'/ || /linux-signed-image-'\$KERNELVERSION'/ || /linux-image-extra-'\$KERNELVERSION'/ { print $2 }'` [ -z "$kernels" ] && { echo "ERROR: no kernels found with version head $KERNELVERSION" >&2 echo "Found kernel images:" >&2 echo "$list" >&2 exit 2 } for kernel in $kernels do echo "sudo apt-get -y --purge remove $kernel" done

Redirect the output of this script to another script, set the new script executable, and then check it for mistakes before executing it!

$ chmod 700 purge-generate.sh
$ purge-generate.sh 3.13.0 >purge.sh
  # check purge.sh before executing it!
$ chmod 700 purge.sh
$ ./purge.sh

The script requires an argument which is the version-head for the kernel(s) to remove. Mind that giving too few numbers (e.g. just "4.15") may include also the current kernel! That's the reason why you need to look at the generated script before executing it.

If everything worked out fine and kernels have been removed, you can update grub to remove obsolete menu items (sometimes already done automatically by apt-get -y --purge):

$ sudo update-grub

This will recreate /boot/grub/grub.cfg which contains the menu items. Further grub configurations you find in /etc/default/grub (modify this before executing update-grub!).

Script Comments

Lines 1-5:
The script first checks that the command line argument is not empty. Mind that the argument may give just the start of the version, so several kernels could be afftected! In case no version match was given, the script outputs its syntax and exits.

Lines 7-13:
The commandline argument is taken into variable KERNELVERSION, which is later injected into the awk script below. But first the kernels are found by listing all installed packages and grep-ing out those that are possibly kernel packages. The `backticks` (command-substitution) put the output of the contained command into variable list.

The awk script reduces the list to those lines that match the KERNELVERSION, and outputs the name of the kernel image, which is the second word, addressed through $2 (which is not variable of the enclosing shell!). Now the variable kernels holds all kernels to remove.

Mind that the double quotes in echo "\$list" are not dispensable. Without them the shell would remove newlines from variable list, and awk would receive just one single line.

Mind also the way how KERNELVERSION is injected into the awk script. The script-text for awk is enclosed into 'single quotes', which end directly before \$KERNELVERSION, and start again directly after. The backslash before $KERNELVERSION is needed due to the enclosing `backticks`, although some shells allow to leave this out.

Lines 15-20:
In case no kernel matched the given version, the script exits and lists existing kernels.

Lines 22-25:
When kernels matched the given version, a command line to remove it gets generated for each of them.

LINUX tar link to zero size

This Blog is about an incident that emptied lots of files and costed me half a day searching for the reason. It is about UNIX tar ("tape archiver").

Circumstances

If you pack an archive with tar, and you unpack it with a tool that doesn't handle hardlinks, you may, under certain circumstances, encounter data loss.

The circumstances happen when the list of files to pack (that you pass to tar) contains a directory that again contains a file that was already listed before. In that case tar packs the file a second time, but this time as link, and with size zero. (It does so although there was no link of any kind in the source file system!) On unpacking with the wrong tool, the link overwrites the real file and truncates it to zero size.

Example

Here is an example directory with files to pack:

tar-test

peter.txt

people.txt

paul.txt

mary.txt

All of peter.txt, paul.txt and mary.txt are normal files. By accident the name of the directory people.txt is similar to that of the files.

Now we want to pack an archive from these files, and we want all *.txt files to be in the archive, so we apply a find command with a wildcard:

cd tar-test tar -cvzf people.tgz `find . -name '*.txt'`

The tar -cvzf command packs an archive (z is for zip-compression), people.tgz is the name of the resulting archive file, and the command-substitution `find . -name '*.txt'` generates the names of files to pack into the archive.

The find command delivers the correct list:

find . -name '*.txt'

./people.txt ./people.txt/mary.txt ./people.txt/paul.txt ./peter.txt

But unfortunately also the directory people.txt matches the pattern '*.txt' and is in that list.

Now when tar processes the directory people.txt, it will pack every file inside also into the archive. Because these files are in the find-list too, the second occurrence of any of them will be stored as link(!) inside the archive.

Look at the result:

tar -tvf people.tgz

drwxrwxrwx root/root 0 2019-04-21 19:19 ./people.txt/ -rwxrwxrwx root/root 11 2019-04-21 19:20 ./people.txt/mary.txt -rwxrwxrwx root/root 10 2019-04-21 19:20 ./people.txt/paul.txt hrwxrwxrwx root/root 0 2019-04-21 19:20 ./people.txt/mary.txt link to ./people.txt/mary.txt hrwxrwxrwx root/root 0 2019-04-21 19:20 ./people.txt/paul.txt link to ./people.txt/paul.txt -rwxrwxrwx root/root 11 2019-04-21 19:20 ./peter.txt

The tar -tvf command lists the archive people.tgz. All files inside the people.txt directory (that also matched the '*.txt' wildcard) were packed twice. We see that all links have size zero (red color).

Mind that unpacking such an archive with tar -xvzf people.tgz works fine, no trace of any links in the resulting file system. You can check this with the ls -l command that lists a link count in 2nd column (red color), and we see that is just 1:

ls -l

-rwxrwxrwx 1 root root 11 Apr 21 19:20 mary.txt -rwxrwxrwx 1 root root 10 Apr 21 19:20 paul.txt

But unpacking with a tool that does not recognize links will do the following:

1. first unpack the non-empty paul.txt file (first because the link depends on it)
2. then unpack the link to paul.txt which has size zero
3. because the empty link has been written over the non-empty file, the file is empty now

Damage done! In case the file is not verified after unpacking, you may not detect the data loss for a long time.

Fix

Here is a way to fix it:

tar -cvzf people.tgz `find . -type f -name '*.txt'`

The find -type f command would find only files, no directories, thus people.txt matching the '*.txt' pattern would not be in its result list.

Resume

Different operating systems have different file systems. Hard and symbolic links exist on UNIX systems, not on WINDOWS. When you use Java tools, you won't be able to create links of any kind, because Java is platform-independent and thus can provide only functionality that is present on all platforms.

Here is a fix for Java (that simply ignores links inside a tar-archive), referring to the com.ice.tar library:

TarEntry e = tarInputStream.getNextEntry();
if (e != null) {
  if (e.getHeader().linkName == null || e.getHeader().linkName.length() <= 0) {
    File created = super.extractEntry(dir, e);
    created.setLastModified(e.getModTime().getTime());
  }
  else {
    System.err.println("Did not extract link: "+e.getName()+" -> "+e.getHeader().linkName);
  }
}

JavaDoc Is Not Deprecated

Modern object-oriented source code doesn't have a big percentage of comments and documentation any more. Recently I measured a project to have just one percent, and these few were texts copied from the specification. So developers do not explain any more why they did it in the way they did it.

I remember times when 25 percent comments in source code was a quality criterion. I'm going to think about the reasons for it in this Blog.

From Domain-Driven Design, Chapter 11, The Flagged Core, p420

A team that uses the code as the sole repository of the model might use comments, maybe structured as JavaDoc, or might use some tool in its development environment. The particular technique doesn't matter, as long as a developer can effortlessly see what is in and what is out of the core domain.

Why Should We Comment?

We should comment because source code is hard to read. It is a translation of some business reality into the jargon of a machine, done through assignments, conditions, loops, and function calls.

I remember a programming test that exposed a 5-line loop where you had to find out what it does, and lots of testees failed. A consequence of what I call hard to read.

The concept behind an implementation gets clear only when you read and understand all involved source code. Developers that have to fix our bugs spend most of their time doing exactly that. We should write comments because the decisions we made during the implementation will not be visible out of the source code.

I remember a unit test, asserted to fail, inside it had a long list of parameters being passed, one of them being wrong, and I had to find out which one is the wrong parameter to be tested. What about marking the one with a comment? Would save a lot of time for readers.

Sustainability is a good reason to write comments. Have mercy with those that come after you. Tell them the harsh truth. Don't scorch the earth. Overcoming the sentence "Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live", we could state a little more positively "Always code as if the girl who ends up maintaining your code is the good looking one living around the corner":-)

Think of global languages full of side effects like CSS. Finding the layout bug and fixing it took you half a day, just for removing one rule. Ok, we can't comment rules that are not there any more, but we should let know others what the problem was, and how it was fixed, right at the place where the bogus rule was. No way around writing a comment (with an issue-number please).

Finally we should write comments for ourselves. People that believe they will remember their code forever are mostly inexperienced newcomers. You need to make it easily readable also for yourself.

Isn't Object-Oriented Code Self-Explanatory?

Surely for the author in the moment it is written. But for others, or viewed a year later, maybe not.

Martin Fowler called comments a sweet smell, because they are always there when something wasn't worked out neatly, with the comment carrying the excuse. But, to be honest, which developer has the time to work out something perfectly? And, in case it is actually an excuse, would we be better off not having that excuse?

Self-explanatory field-, method- and class-names can't always express everything that is important for the caller. In most cases they would simply be too long when telling everything that happens within them, and using abbreviations instead of long names would make it even worse. That's what JavaDoc was made for.

The core problem is that technical complexity is so overwhelming sometimes that we run out of words to describe it precisely.

Can We Tackle Complexity by Header Comments?

Yes we can. The price though is duplication of semantic, from code into human readable text.

Source code is a big tree (or graph) of classes and methods calling other classes and methods. In case you need to call a public method of another class that was named process() or the like, you would have to find out what it does by reading all the source code below it.
Now imagine that this topmost method has some JavaDoc attached. If you configured your IDE to display JavaDoc on mouse-over, you could dare to trust that quick information. Takes half a minute to understand, versus one hour studying private sub-calls. Promotion of encapsulation. I believe that Java wouldn't have become so popular if its libraries weren't javadoc'ed sufficiently already in 1995.

Comments can tackle complexity also in a psychological way. If you write the header comment before you implement the method, you may find out that your first plan about it will not work. Designing first saves you the time you will need to debug fast-built but failing implementations. You start to design while you think about the real intent, especially when you describe responsibilities.

An even higher level is writing the design into the issue you are working on. In that case a colleague could review it and give valuable hints. You could finally copy the ready-made concept into a JavaDoc class header.

Header and Inline Comments

Header comments are on public classes, methods and constant fields. Inline comments are inside classes and methods, explaining details.

1. We should explain the concepts we followed during implementation. There is no better place than header comments for documenting concepts.


2. In inline comments you should not comment what you are doing, but why you are doing it. This is what readers will ask: Why is this method invocation here and not at the end of the block?

How to JavaDoc

Remember CRC cards, one of the earliest OO software design methods: Class-Responsibility-Collaboration. Put the responsibility and collaboration of the class into its header comment!

Following is a piece of code that I would consider to be documented sufficiently.

/**
 * Responsibility of this class is to show that JavaDoc is not useless.
 * Context is a project with thousands of classes with very general names.
 *
 * @author fritzthecat 2019-04-16
 */
public class JavaDocIsNotDeprecated
{
    /** Publics always should be javadoc'ed. */
    public static final String AGREE = "More comments!";

    private int count;  // privates should have good names to be self-explanatory

    /**
     * Save our time by documenting!
     * @param comment the text to write as comment to stdout.
     * @return the instance for more comments.
     */
    public JavaDocIsNotDeprecated writeComment(String comment) {
        System.out.println(comment);  // use out because err is not platform-independent
        return this;
    }

}

All publics should be documented, because they are visible to the outer world. Protected also need to be, although you should avoid protected fields (results in weak encapsulation). Inline comments explain reasons and intents.

Resume

Comments and documentation is considered to be useless extra work. Useless because words can not describe the complexity we saw. Extra work because it is duplication of code that doesn't auto-update itself.

It was also the guy with the "sweet smell" that told us to "write code for humans, not computers". So, can we find method names that will inform humans about what is going on? Or would we like to express it in natural language in a JavaDoc comment?

Security versus Freedom

The Internet provides great freedom, and we already got used to it. Now the teller is presenting the bill. Security costs money. Without security you may be blackmailed for ransom, your user's data may be stolen, or your Internet service may be blocked. So, whom would you prefer to pay, the hacker or the security expert?

I recently had the opportunity to attend a presentation about web security. This has become a big business. You pay a specialist to do a "pen test" - no, it's not about pencils, it's about penetration of your IT company. Here are some notes about this meeting. Lots of new acronyms to learn.

Hacker's Operating System

Kali LINUX

A Debian LINUX already equipped with most tools that a hacker or penetration tester will want to use. Started 2012.

Hacker's Tool Suite

Burp Suite

A graphical user interface to work out a hack. Provides screenshots of all pages of a site, easy HTTP request manipulations and resends. You can do a lot of things with this tool, but you must study it.

Penetration Test Tools

Nmap	Open source, started 1998. Provides host discovery, service- and operating-system detection, ....
Nessus	Started 1998, commercial having version 8, but version 2 is still open source and maintained. Finds open ports, executes brute force password detection attacks, can launch Denial of Service attacks.
Metasploit	Open source, started 2003. Specializes on exploits through operating-system-specific vulnerabilities.

Mitigation on Software Level

OWASP	OWASP = Open Web Application Security Project. Online community that produces freely-available articles, methodologies, documentation, tools, and technologies.
CVE	CVE = Common Vulnerability Exposure.

Mitigation on Network Level

ZAP	ZAP = Zed Attack Proxy. An OWASP tool. Used as a proxy server it allows to manipulate all of the HTTP/HTTPS traffic that passes through. Intended to be used by both those new to application security as well as professional penetration testers.
CDN	CDN = Content Delivery Network. Cloudflare or MaxCDN are commercial caching- and security-services that may defend Distributed Denial of Service attacks.
IDS, IPS	IDS = Intrusion Detection System, IPS = Intrusion Prevention System. Sits behind the firewall and provides analysis about dangerous content (IDS), or filters it out (IPS).
WAF	WAF = Web Application Firewall. Can prevent attacks through SQL injection, cross-site-scripting (XSS), file inclusion, and security misconfigurations.

Others

haveibeenpwned	Has one of your e-mail accounts been hacked?
yasni	What the Internet knows about you.
aircrack-ng	WIFI (wireless network) security checks.
Open Bug Bounty	Cross-site-scripting (XSS) vulnerabilities.
Beef Project	Specializes on browser vulnerabilities.
Crackstation	Free password hash cracker. Uses massive pre-computed lookup tables to turn password hashes back into passwords.
Shodan	Commercial website, a search service for the "Internet of Things" (including all kinds of databases). Called "The Hacker's Google".
eyewitness	A headless browser producing screenshots.
SecurityTrails	The largest collection of IP addresses, domain names and WHOIS data.
NIST	NIST = National Institute of Standards and Technology. Computer security on organisational level.

Docker Basics

Docker is for deployment. It is a virtualization at operating-system level. When you pack your application into a docker image and run that image, the application will believe that it has the whole computer on its own. All dependencies will be in that image, as configured in a Dockerfile of your application together with build- and runtime-environment variables. You don't depend on any software being installed on your deployment target.

Virtualization is what virtual machines did for us until now, but docker can do that more lightweight and much faster. Originally it was available on LINUX only, because the LINUX kernel can somehow "duplicate" itself (cgroups, namespaces), which feels like more than one operating-system "instances" running on the same hardware. For WINDOWS 10 this is available as "Hyper V" feature, you must activate virtualization in the BIOS of your computer; alternatively run a LINUX VM that hosts the docker installation for WINDOWS. Such said you will understand that, instead of a VM, docker has to be installed on the machine where you want to run your docker image.

You can find all what's in this Blog on the docker homepage. Lots of docker images are available freely on the internet.

Mind that ....

• docker aims at the cloud, the packed app may be expected to be a 12-factor app


• docker doesn't provide operating-system-independence, the target operating-system must support the packed app


• all data that a dockerized app writes to the file system it runs upon will be lost when its container gets removed


• two Java apps packed into two docker images will carry two JREs and not share them


• all dockerized Java apps will see the same number of CPU cores and the same memory amount, so if they scale themselves (like a JVM does), they may overuse system resources (switch to Java 9 or newer and use JVM options -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap).

Build Image, Run Containers

The image is what you build. The container is what you run.

The image won't change, it's the template for the container, while the container is a process that can take on states. You can instantiate many containers from one image. Containers are still present after they terminated, unless they were created using the --rm flag, like in docker run --rm imagename.

Useful Commands

You can manage your docker platform by lots of commands that all start with "docker".

Purpose	Command
List all images installed on computer	docker images docker image ls
List all running containers on computer	docker ps docker container ls
List all running and stopped containers on computer	docker ps -a docker container ls -a
Instantiate a container from image "hello-world", with download if the image is not present	docker run hello-world
Look at the logs of container "mysql"	docker logs --follow mysql
Terminate the container with identity "mysql"	docker stop mysql
Start again the stopped container with identity "mysql", using the same file system	docker start mysql
Stop and start again the running container with identity "mysql"	docker restart mysql
Remove a container "hello-world"	docker rm hello-world
Remove an image "hello-world"	docker rmi hello-world
Start an interactive shell (bash) inside a running container "mysql"	docker exec -it mysql bash
Copy a file /opt/mysql/my.ini from a container "mysql" into current directory (".")	sudo docker cp mysql:/opt/mysql/my.ini .
Display the layers inside the image "hello-world"	docker history hello-world
Create an image out of the app in current directory where a Dockerfile exists	docker build .
Remove all stopped containers, unused images and other resources	docker system prune -a
Remove all images not connected to a container	docker image prune -a

Dockerfile Keywords

A Dockerfile normally is placed in the root directory of the project that needs to be packed into a docker image. It's like a C makefile, or a Java/Maven pom.xml, but it has its own syntax. Following is a short reference of some important keywords used in such a file.

Keyword	Meaning
#	Exclusively at line start, opens a comment or a parser-directive like "# directive=... ", "# syntax=...", "# escape=..."
FROM	The docker image this app builds on. A Java app would put a JRE here. Dependencies of dependencies need not to be listed. A valid Dockerfile always starts with FROM.
ARG	Buildtime variable, optionally with default-value. Can be set or overwritten by an option in the docker build command. An ENV- or ARG-variable XXX can be used inside a Dockerfile via $XXX or ${XXX}.
ENV	Runtime variable for OS-environment of the contained app. Can be overwritten by an option in the docker run command.
LABEL	Metadata, can be recalled by docker inspect.
EXPOSE	The port where the contained app listens, defaults to a TCP type port.
VOLUME	Creates a directory (or mount point) shared with the outside world that will not be removed when the container gets removed.
WORKDIR	Sets the working directory for all RUN, CMD, ENTRYPOINT, COPY, ADD statements following in the Dockerfile. In case WORKDIR doesn't exist, it will be created.
USER	Sets the system-user for all RUN, CMD, ENTRYPOINT statements following in the Dockerfile.
COPY	Buildtime, copies files from file system into the image to build.
ADD	Buildtime, copies files from file system or URLs from the network into the image to build.
RUN	Buildtime, executes the command to the right of RUN.
CMD	Runtime, starts the contained app on docker run. Only one CMD statement is possible in a Dockerfile.

Mind that setting the image's name inside a Dockerfile is not supported!

Layers and Gotchas

A Dockerfile like this one

FROM debian:jessie
ADD large_file /var/www/large_file
RUN chown www-data /var/www/large_file
RUN chmod 756 /var/www/large_file

that packs a large_file of 1 GB size and then changes its access rights will lead to a 3 GB image with following layers:

IMAGE          CREATED          CREATED BY                                      SIZE        COMMENT
49b4a4ea228a   36 seconds ago   /bin/sh -c chmod 756 /var/www/large_file        1.074 GB
09d77316932b   2 minutes ago    /bin/sh -c chown www-data /var/www/large_file   1.074 GB
7adb7c72c3ef   2 minutes ago    /bin/sh -c #(nop) ADD file:a86f6dedfb4ba54972   1.074 GB
f50f9524513f   8 weeks ago      /bin/sh -c #(nop) CMD ["/bin/bash"]             0 B
<missing>      8 weeks ago      /bin/sh -c #(nop) ADD file:b5391cb13172fb513d   125.1 MB

That means, every build step like FROM, RUN, COPY, ADD creates a new layer, and files referenced in the commands executed there will be copied always newly into the image.

This also affects final cleanups of left-over files, they would be deleted in the topmost layer only.
Docker observes the file system and notices any change after a build step (resulting in a new layer), but it doesn't associate steps and draw conclusions.

Workaround:

Commands that change the file system should be linked together by the '&&' operator to just one command:

RUN apt-get update \
    && apt-get install -y vim \
    && rm -rf /var/lib/apt

A chown could be done by a preparing "RUN usermod -u 1000 www-data".
Alternatively the container can be started by a script that provides all necessary circumstances:

docker run hello-world --entrypoint=/bin/myscript.sh

In the latter case the /bin/myscript.sh should be inside the docker image.