How to Implement and Debug a Java Doclet

If you've never heard about Doclet, this is a JDK library that enables you to write your own variant of the javadoc tool that comes bundled with the JDK. This tool is for generating HTML pages from class-, field- and method- /** comments */ that were written in Java sources by developers.

Doclet has been reworked for modular Java, and its use was documented newly. Mind that some Java 8 documentation still refers to the old version (which says a lot about javadoc popularity). The best time of Doclet was before annotations were introduced with Java 1.6 in 2007, as it was used as replacement for annotations.

The interest in documentation never was great in developer circles, being called a "sweet code smell", and everybody was happy that he/she could believe in self-explanatory source code (which I regard to be a myth, but myths are more popular than reality). Nevertheless we all want autocomplete-tooltips in our IDE, showing us the documentation of the API that we need to call (because there is very few self-explanatory code in this world:-).

However, if you are interested in writing a Doclet, here is a description of how you can implement and launch it in your IDE's debugger. A Doclet normally is launched by -doclet and -docletpath options in command line of the javadoc tool, but it's not possible to debug it then, so we need a solution.

Overview

What we need to implement:

1. A static Java launcher of the javadoc tool, with the options to use our doclet. This class can then be started in "Run" or "Debug" mode of your IDE.


2. The Doclet interface. The resulting class can be used with the -doclet option in command line of the javadoc tool.


3. The ElementVisitor interface that is accepted by the "elements" returned from DocletEnvironment.html#getSpecifiedElements() in the run() implementation of the Doclet interface.
   In this article you will find just an abstraction of that visitor (called ModelBuilder), because it is up to you to decide what your doclet should do.

Launcher and Doclet

I implemented a parameterizable launcher and the Doclet interface in just one class. This must be run from another class implementing a runnable main() method. You find an example at the end of this chapter.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80

import java.util.ArrayList; import java.util.Collections; import java.util.List; import java.util.Locale; import java.util.Set; import javax.lang.model.SourceVersion; import javax.lang.model.element.Element; import javax.tools.DocumentationTool; import javax.tools.ToolProvider; import jdk.javadoc.doclet.Doclet; import jdk.javadoc.doclet.DocletEnvironment; import jdk.javadoc.doclet.Reporter; /** * A javadoc launcher that implements the Doclet interface. */ public class DocletLauncher implements Doclet { /** * Launcher that calls javadoc. * @param docletClass the doclet class to use, value for "-doclet" option. * @param DOCLET_CLASSPATH the path where the doclet resides, value for "-docletpath". * @param SOURCE_BASEPATH the path where the sources reside, prefix for all sourceFiles. * @param sourceFiles Java sources to process. */ public static void startDoclet( String DOCLET_CLASSPATH, String SOURCE_BASEPATH, String[] sourceFiles) { // for every file, call javadoc for (String sourceFile : sourceFiles) { if (sourceFile.endsWith(".java") == false) { if (sourceFile.contains(".")) sourceFile = sourceFile.replace('.', '/'); sourceFile = sourceFile+".java"; } // use a custom stream to get file names and line numbers final String[] docletArguments = new String[] { "-doclet", DocletLauncher.class.getName(), "-docletpath", DOCLET_CLASSPATH, SOURCE_BASEPATH+"/"+sourceFile }; final DocumentationTool javaDoc = ToolProvider.getSystemDocumentationTool(); javaDoc.run(null, null, null, docletArguments); // in, out, err } } /** Called by javadoc for source visitation. */ @Override public boolean run(DocletEnvironment environment) { for (Element element : environment.getSpecifiedElements()) { final ModelBuilder modelBuilder = new ModelBuilder(environment.getDocTrees()); element.accept(modelBuilder, null); } return true; } @Override public void init(Locale locale, Reporter reporter) { } @Override public String getName() { return getClass().getSimpleName(); } @Override public Set<? extends Option> getSupportedOptions() { return Collections.emptySet(); } @Override public SourceVersion getSupportedSourceVersion() { return SourceVersion.latest(); } }

The startDoclet() method on line 28 can be run from any class main() with parameters according to your development environment (IDE). For a simple Eclipse Java project (non-Maven), the DOCLET_CLASSPATH would be bin, for an Eclipse Maven module, target/classes. The SOURCE_BASEPATH must point to some path where the Java source files to analyze are below. Mind that you need to add all compile-dependencies to your CLASSPATH when scanning sources outside the module where your doclet resides, because javadoc uses the javac compiler!

On line 34, all source files given as arguments are looped and passed to the javadoc tool, one by one. On line 36, the arguments are converted into file system paths when given as fully-qualified class names.

Line 42 builds together the arguments to be passed to javadoc. Every source file is placed under the given SOURCE_BASEPATH.

Line 47 and 48 finally run the javadoc tool. This will trigger, for every source file in loop, the run() method on line 54. Mind that we skip from static usage to the Doclet instance here.

The run() method on line 55 loops all "specified elements" in given source file. Mind that if you have a public inner interface or class in your source file, that will be a separate "specified element", although it also will be contained in the main element.

The ModelBuilder on line 56 is not a JDK class, it is the ElementVisitor implementation specific to the doclet we provide here. You can see its abstraction below. The DocTrees object that is passed to it is a utility needed to retrieve comments, file paths and line numbers of the scanned source.

All other methods do not matter very much, they were not called by javadoc for a single Java file.

---

Example for a class that calls the launcher, which you would run in debug-mode from your IDE:

public class StartDoclet
{
    public static void main(String[] args) {
        final String DOCLET_CLASSPATH = "bin";    // or "target/classes"
        final String SOURCE_BASEPATH = "src";    // or "src/main/java"
        //final String SOURCE_BASEPATH = "../my_module/src/main/java/";
        
        DocletLauncher.startDoclet(
                DOCLET_CLASSPATH, 
                SOURCE_BASEPATH, 
                args);
    }
}

If you run StartDoclet from another module than where the source to scan is, you would pass ../my_module/src/main/java/ as SOURCE_BASEPATH, in case the source is in a parallel Maven-module called "my_module". It is important to also take that other project and all its dependencies into the CLASSPATH when running the StartDoclet, because javadoc uses the javac compiler, and that would fail when trying to compile some source without having its dependencies. For Eclipse, you can do that on the "Dependencies" tab in "Run/Debug Configuration".

Alternatively you can place a StartDoclet class in every module where you intend to scan source. In that case, each of these modules must refer to the DocletLauncher module, thus you would revert the dependency.

Element Visitor

Writing the Doclet was easy, here comes the hard part. It took me some time to find out all the details of the doclet API, for example it was really hard to get the source's file path, or the line number of some element.

The class below is an abstraction of what you need to implement for Java source visitation. It translates the quite complex doclet visitation to a simpler, supporting inner class nesting. All public visitXXX methods belong to the Doclet interface, all protected abstract visitXXX methods need to be overridden by a concrete ModelBuilder that would produce HTML code. Do not confuse these two method groups.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161

import java.util.HashSet; import java.util.List; import java.util.Set; import javax.lang.model.element.Element; import javax.lang.model.element.ElementVisitor; import javax.lang.model.element.ExecutableElement; import javax.lang.model.element.PackageElement; import javax.lang.model.element.TypeElement; import javax.lang.model.element.TypeParameterElement; import javax.lang.model.element.VariableElement; import javax.tools.JavaFileObject; import com.sun.source.doctree.DocCommentTree; import com.sun.source.doctree.DocTree; import com.sun.source.tree.CompilationUnitTree; import com.sun.source.tree.LineMap; import com.sun.source.util.DocSourcePositions; import com.sun.source.util.DocTrees; import com.sun.source.util.TreePath; /** * To be overridden for interpretation of one Java source file. */ public abstract class AbstractModelBuilder implements ElementVisitor<Void,Object> { protected final DocTrees docTrees; private final Set<Integer> lineNumbers = new HashSet<>(); public AbstractModelBuilder(DocTrees docTrees) { this.docTrees = docTrees; } // START interface ElementVisitor @Override public final Void visitType(TypeElement element, Object parameter) { if (lineNumbers.isEmpty()) if (visitFile(filePath(element)) == false) return null; final int line = fileLine(element); lineNumbers.add(line); // avoids receiving no-arg constructor created by compiler final String className = getName(element); visitClass(className, line, getComment(element), element); // to achieve nesting of inner classes, explicitly loop enclosed elements for (Element enclosed : element.getEnclosedElements()) enclosed.accept(this, parameter); visitEndOfClass(className, element); // terminate current inner class return null; } @Override public final Void visitVariable(VariableElement element, Object parameter) { visitFieldOrMethod(element, false); return null; } @Override public final Void visitExecutable(ExecutableElement element, Object parameter) { visitFieldOrMethod(element, true); return null; } @Override public final Void visit(Element e, Object p) { return null; } @Override public final Void visitPackage(PackageElement e, Object p) { return null; } @Override public final Void visitTypeParameter(TypeParameterElement e, Object p) { return null; } @Override public final Void visitUnknown(Element e, Object p) { return null; } // END interface ElementVisitor // START to be implemented by sub-classes /** * Sub-classes can process given source file path. By default this returns true. * @return true when processing of this file should continue, false to ignore it. */ protected boolean visitFile(String filePath) { return true; } /** Sub-classes must process given top-level or inner class. */ protected abstract void visitClass(String name, int line, String comment, TypeElement e); /** Sub-classes must process given field. */ protected abstract void visitField(String name, int line, String comment, VariableElement e); /** Sub-classes must process given method. */ protected abstract void visitMethod(String name, int line, String comment, ExecutableElement e); /** The current class ends now. */ protected abstract void visitEndOfClass(String name, TypeElement e); // END to be implemented by sub-classes private final String filePath(TypeElement e) { final CompilationUnitTree compilationUnit = docTrees.getPath(e).getCompilationUnit(); final JavaFileObject fileObject = compilationUnit.getSourceFile(); return fileObject.getName(); } private void visitFieldOrMethod(Element e, boolean isMethod) { final int line = fileLine(e); // avoid multiple visitations due to explicitly looping enclosed elements if (lineNumbers.contains(line) == false) { lineNumbers.add(line); if (isMethod) visitMethod(getName(e), line, getComment(e), (ExecutableElement) e); else visitField(getName(e), line, getComment(e), (VariableElement) e); } } private String getName(Element e) { return (e instanceof TypeElement) ? e.toString() : e.getSimpleName().toString(); } private String getComment(Element e) { final DocCommentTree docCommentTree = docTrees.getDocCommentTree(e); if (docCommentTree == null) return null; final StringBuilder sb = new StringBuilder(); final List<? extends DocTree> fullBody = docCommentTree.getFullBody(); for (final DocTree commentFragment : fullBody) { final String text = commentFragment.toString().trim(); if (sb.length() <= 0) sb.append(text); else sb.append(" "+text); } final String result = sb.toString(); return result.isBlank() ? null : result; } private final int fileLine(Element e) { final TreePath path = docTrees.getPath(e); final CompilationUnitTree compilationUnit = docTrees.getPath(e).getCompilationUnit(); final LineMap lineMap = compilationUnit.getLineMap(); final DocSourcePositions spos = docTrees.getSourcePositions(); final long startPos = spos.getStartPosition(compilationUnit, path.getLeaf()); return (int) lineMap.getLineNumber(startPos); } }

Besides the java.lang.model imports we also have com.sun.source now (don't ask me if this is a "legal" import), needed for DocTrees helpers.

Implementing ElementVisitor<Void,Object> on line 25 means that the visit-methods won't return anything (Void), and their second parameter will be of type Object (actually this is not used here, it would be the second parameter you pass to accept() in DocletLauncher). The constructor on line 30 stores the DocTrees utility, coming from the caller's DocletEnvironment.

Line 37 - 84 contain all implementations of the doclet ElementVisitor interface. Only three of them do something, because AbstractModelBuilder just catches classes, (visitType(), also all inner classes go here), fields (visitVariable()) and methods (visitExecutable()). The first visitType() call also exposes the source file path.

The visitType() implementation explicitly iterates over the enclosed elements on line 49. This technique requires to avoid duplicate calls, because all elements would be visited anyway also without this explicit loop. By storing line numbers we avoid duplicate calls, and achieve nesting of inner classes. The visitEndOfClass() call on line 52 notifies about exiting an inner class.

Both visitVariable() and visitExecutable() also need to comply with the line number management. To do this once-and-only-once, both call the same procedure on line 119, with a flag.

All other methods from line 70 - 84 are not called when scanning a single Java source file. Line 94 - 108 contain the methods you need to implement when you want to use this class for your doclet. Maybe you should add data types, and a list of parameters to method visiting. You can retrieve such informations from the Element derivations VariableElement and ExecutableElement.

On line 113 the internals start. It looks like we need to know about "compilation unit" to get the source file path on line 114, and the line numbers on line 153. Both make use of the DocTrees utlitity that was given to constructor.

Line 132 shows how to get the name of a class, field or method. All can be done with "simple name". The toString() method should not carry application semantic, but here it was done so. (Not the only inconsistency, why was field called "variable", or method "executable"?)

For getting the comment of some element on line 135 you would need a dedicated DocTreeVisitor that you can pass to every single DocTree item in list. I tried to make it simpler, again with the infamous toString() call. Complex comments may not look good, and @tags are not contained.

Resume

I hope this helps in case you need to write a doclet. The doclet API was made for experts, not for the masses. For me it was a frustrating experience, because I just wanted to achieve some source folding as it can be seen in my recent articles, and it took me days to come to an end. Let's hope that javadoc will become more popular again in the future, we need it!