Q:

Grails 2.5.0 not using my lib/ jars when deployed to tomcat

I had to compile a custom version of the flying saucer jars used by the rendering plugin to get around a problem with rendering and bootstrap css. It works fine locally...I can render a gsp to PDF. If I don't include my custom jars in the lib directory, I get the following error:
com.lowagie.text.pdf.BaseFont.getCharBBox(C)[I. Stacktrace follows:
org.codehaus.groovy.grails.web.servlet.mvc.exceptions.ControllerExecutionException: Executing action [invoiceAll] of controller [myapp.ReportsController]  caused exception: Runtime error executing action
    at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
    at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
    at grails.plugin.springsecurity.web.filter.GrailsAnonymousAuthenticationFilter.doFilter(GrailsAnonymousAuthenticationFilter.java:53)
    at grails.plugin.springsecurity.web.authentication.RequestHolderAuthenticationFilter.doFilter(RequestHolderAuthenticationFilter.java:49)
    at grails.plugin.springsecurity.web.authentication.logout.MutableLogoutFilter.doFilter(MutableLogoutFilter.java:82)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.codehaus.groovy.grails.web.servlet.mvc.exceptions.ControllerExecutionException: Runtime error executing action
    ... 8 more
Caused by: java.lang.reflect.InvocationTargetException
    ... 8 more
Caused by: java.lang.NoSuchMethodError: com.lowagie.text.pdf.BaseFont.getCharBBox(C)[I
    at org.xhtmlrenderer.pdf.ITextFontResolver$FontDescription.setMetricDefaults(ITextFontResolver.java:679)
    at org.xhtmlrenderer.pdf.ITextFontResolver$FontDescription.<init>(ITextFontResolver.java:610)
    at org.xhtmlrenderer.pdf.ITextFontResolver.addCourier(ITextFontResolver.java:410)
    at org.xhtmlrenderer.pdf.ITextFontResolver.createInitialFontMap(ITextFontResolver.java:390)
    at org.xhtmlrenderer.pdf.ITextFontResolver.<init>(ITextFontResolver.java:52)
    at org.xhtmlrenderer.pdf.ITextRenderer.<init>(ITextRenderer.java:115)
    at org.xhtmlrenderer.pdf.ITextRenderer.<init>(ITextRenderer.java:102)
    at grails.plugin.rendering.pdf.PdfRenderingService.doRender(PdfRenderingService.groovy:34)
    at grails.plugin.rendering.RenderingService.render(RenderingService.groovy:43)
    at grails.plugin.rendering.RenderingService.render(RenderingService.groovy:37)
    at grails.plugin.rendering.RenderingService.render(RenderingService.groovy:35)
    at grails.plugin.rendering.RenderingService.render(RenderingService.groovy:65)
    at RenderingGrailsPlugin$_closure3.doCall(RenderingGrailsPlugin.groovy:59)
    at myapp.ReportsController.invoiceAll(ReportsController.groovy:46)
    ... 8 more

So, doing this locally works fine. lsof shows my jars in use by the grails java process:
java    26951 myuser  mem    REG             253,14    105122  531524 /home/myuser/sworkspace/groovy/tls_billing/lib/flying-saucer-pdf-9.0.1-myapp.jar
java    26951 myuser  mem    REG             253,14   1106329  528159 /home/myuser/sworkspace/groovy/tls_billing/lib/flying-saucer-core-9.0.1-myapp.jar
java    26951 myuser  mem    REG             253,14    105172  531525 /home/myuser/sworkspace/groovy/tls_billing/lib/flying-saucer-pdf-itext5-9.0.1-myapp.jar
java    26951 myuser   68r   REG             253,14    105172  531525 /home/myuser/sworkspace/groovy/tls_billing/lib/flying-saucer-pdf-itext5-9.0.1-myapp.jar
java    26951 myuser   69r   REG             253,14   1106329  528159 /home/myuser/sworkspace/groovy/tls_billing/lib/flying-saucer-core-9.0.1-myapp.jar
java    26951 myuser   75r   REG             253,14    105122  531524 /home/myuser/sworkspace/groovy/tls_billing/lib/flying-saucer-pdf-9.0.1-myapp.jar

Just to be safe, I also sha1sum'd the jars locally:
2c135a987892978d670cd73e789fa3337d50e7ff  lib/flying-saucer-core-9.0.1-myapp.jar
b774052563a12fc212370a1c1a8a71ea5b8c0e87  lib/flying-saucer-pdf-9.0.1-myapp.jar
64c01c5f3425b0723b0d2e4a034fe9f23e3e1720  lib/flying-saucer-pdf-itext5-9.0.1-myapp.jar

So, after "grails war" and then deploying to a remote Tomcat server, I get the following error:
2015-07-15 16:56:20,221 [http-bio-80-exec-1] ERROR StackTrace  - Full Stack Trace:
java.lang.reflect.InvocationTargetException
        at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
        at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
        at grails.plugin.springsecurity.web.filter.GrailsAnonymousAuthenticationFilter.doFilter(GrailsAnonymousAuthenticationFilter.java:53)
        at grails.plugin.springsecurity.web.authentication.RequestHolderAuthenticationFilter.doFilter(RequestHolderAuthenticationFilter.java:49)
        at grails.plugin.springsecurity.web.authentication.logout.MutableLogoutFilter.doFilter(MutableLogoutFilter.java:82)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoSuchMethodError: com.lowagie.text.pdf.BaseFont.getCharBBox(C)[I
        at org.xhtmlrenderer.pdf.ITextFontResolver$FontDescription.setMetricDefaults(ITextFontResolver.java:679)
        at org.xhtmlrenderer.pdf.ITextFontResolver$FontDescription.<init>(ITextFontResolver.java:610)
        at org.xhtmlrenderer.pdf.ITextFontResolver.addCourier(ITextFontResolver.java:410)
        at org.xhtmlrenderer.pdf.ITextFontResolver.createInitialFontMap(ITextFontResolver.java:390)
        at org.xhtmlrenderer.pdf.ITextFontResolver.<init>(ITextFontResolver.java:52)
        at org.xhtmlrenderer.pdf.ITextRenderer.<init>(ITextRenderer.java:115)
        at org.xhtmlrenderer.pdf.ITextRenderer.<init>(ITextRenderer.java:102)
        at grails.plugin.rendering.pdf.PdfRenderingService.doRender(PdfRenderingService.groovy:34)
        at grails.plugin.rendering.RenderingService.render(RenderingService.groovy:43)
        at grails.plugin.rendering.RenderingService.render(RenderingService.groovy:37)
        at grails.plugin.rendering.RenderingService.render(RenderingService.groovy:35)
        at grails.plugin.rendering.RenderingService.render(RenderingService.groovy:65)
        at RenderingGrailsPlugin$_closure3.doCall(RenderingGrailsPlugin.groovy:59)
        at myapp.ReportsController.invoiceAll(ReportsController.groovy:46)
        ... 8 more
2015-07-15 16:56:20,221 [http-bio-80-exec-1] ERROR StackTrace  - Full Stack Trace:
java.lang.NoSuchMethodError: com.lowagie.text.pdf.BaseFont.getCharBBox(C)[I
        at org.xhtmlrenderer.pdf.ITextFontResolver$FontDescription.setMetricDefaults(ITextFontResolver.java:679)
        at org.xhtmlrenderer.pdf.ITextFontResolver$FontDescription.<init>(ITextFontResolver.java:610)
        at org.xhtmlrenderer.pdf.ITextFontResolver.addCourier(ITextFontResolver.java:410)
        at org.xhtmlrenderer.pdf.ITextFontResolver.createInitialFontMap(ITextFontResolver.java:390)
        at org.xhtmlrenderer.pdf.ITextFontResolver.<init>(ITextFontResolver.java:52)
        at org.xhtmlrenderer.pdf.ITextRenderer.<init>(ITextRenderer.java:115)
        at org.xhtmlrenderer.pdf.ITextRenderer.<init>(ITextRenderer.java:102)
        at grails.plugin.rendering.pdf.PdfRenderingService.doRender(PdfRenderingService.groovy:34)
        at grails.plugin.rendering.RenderingService.render(RenderingService.groovy:43)
        at grails.plugin.rendering.RenderingService.render(RenderingService.groovy:37)
        at grails.plugin.rendering.RenderingService.render(RenderingService.groovy:35)
        at grails.plugin.rendering.RenderingService.render(RenderingService.groovy:65)
        at RenderingGrailsPlugin$_closure3.doCall(RenderingGrailsPlugin.groovy:59)
        at myapp.ReportsController.invoiceAll(ReportsController.groovy:46)
        at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
        at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
        at grails.plugin.springsecurity.web.filter.GrailsAnonymousAuthenticationFilter.doFilter(GrailsAnonymousAuthenticationFilter.java:53)
        at grails.plugin.springsecurity.web.authentication.RequestHolderAuthenticationFilter.doFilter(RequestHolderAuthenticationFilter.java:49)
        at grails.plugin.springsecurity.web.authentication.logout.MutableLogoutFilter.doFilter(MutableLogoutFilter.java:82)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
2015-07-15 16:56:20,221 [http-bio-80-exec-1] ERROR StackTrace  - Full Stack Trace:
org.codehaus.groovy.grails.web.servlet.mvc.exceptions.ControllerExecutionException: Executing action [invoiceAll] of controller [myapp.ReportsController]  caused exception: Runtime error executing action
        at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
        at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
        at grails.plugin.springsecurity.web.filter.GrailsAnonymousAuthenticationFilter.doFilter(GrailsAnonymousAuthenticationFilter.java:53)
        at grails.plugin.springsecurity.web.authentication.RequestHolderAuthenticationFilter.doFilter(RequestHolderAuthenticationFilter.java:49)
        at grails.plugin.springsecurity.web.authentication.logout.MutableLogoutFilter.doFilter(MutableLogoutFilter.java:82)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.codehaus.groovy.grails.web.servlet.mvc.exceptions.ControllerExecutionException: Runtime error executing action
        ... 8 more
Caused by: java.lang.reflect.InvocationTargetException
        ... 8 more
Caused by: java.lang.NoSuchMethodError: com.lowagie.text.pdf.BaseFont.getCharBBox(C)[I
        at org.xhtmlrenderer.pdf.ITextFontResolver$FontDescription.setMetricDefaults(ITextFontResolver.java:679)
        at org.xhtmlrenderer.pdf.ITextFontResolver$FontDescription.<init>(ITextFontResolver.java:610)
        at org.xhtmlrenderer.pdf.ITextFontResolver.addCourier(ITextFontResolver.java:410)
        at org.xhtmlrenderer.pdf.ITextFontResolver.createInitialFontMap(ITextFontResolver.java:390)
        at org.xhtmlrenderer.pdf.ITextFontResolver.<init>(ITextFontResolver.java:52)
        at org.xhtmlrenderer.pdf.ITextRenderer.<init>(ITextRenderer.java:115)
        at org.xhtmlrenderer.pdf.ITextRenderer.<init>(ITextRenderer.java:102)
        at grails.plugin.rendering.pdf.PdfRenderingService.doRender(PdfRenderingService.groovy:34)
        at grails.plugin.rendering.RenderingService.render(RenderingService.groovy:43)
        at grails.plugin.rendering.RenderingService.render(RenderingService.groovy:37)
        at grails.plugin.rendering.RenderingService.render(RenderingService.groovy:35)
        at grails.plugin.rendering.RenderingService.render(RenderingService.groovy:65)
        at RenderingGrailsPlugin$_closure3.doCall(RenderingGrailsPlugin.groovy:59)
        at myapp.ReportsController.invoiceAll(ReportsController.groovy:46)
        ... 8 more

Just to be certain, I cd'd into /opt/tomcat/webapps/ROOT/WEB-INF/lib (where my app is deployed) and ran:
$ sha1sum flying-saucer-*
2c135a987892978d670cd73e789fa3337d50e7ff  flying-saucer-core-9.0.1-myapp.jar
b774052563a12fc212370a1c1a8a71ea5b8c0e87  flying-saucer-pdf-9.0.1-myapp.jar
64c01c5f3425b0723b0d2e4a034fe9f23e3e1720  flying-saucer-pdf-itext5-9.0.1-myapp.jar

Matches up. However, is the app actually holding file handles open to the jars?
At first, after I hadn't loaded the URL for a bit, it didn't appear to be:
$ sudo lsof | grep flying | wc -l 
0

I assume Tomcat spins down threads though, because after visiting the URL which uses the jar (and causes the exception) again, I get hits:
$ sudo lsof | grep flying

F/lib/flying-saucer-pdf-itext5-9.0.1-myapp.jar
java      24487 24542        root  mem       REG              254,0  1106329     268857 /opt/apache-tomcat-7.0.61/webapps/ROOT/WEB-INF/lib/flying-saucer-core-9.0.1-myapp.jar
java      24487 24542        root  mem       REG              254,0   105122     268824 /opt/apache-tomcat-7.0.61/webapps/ROOT/WEB-INF/lib/flying-saucer-pdf-9.0.1-myapp.jar
java      24487 24542        root   93r      REG              254,0  1106329     268857 /opt/apache-tomcat-7.0.61/webapps/ROOT/WEB-INF/lib/flying-saucer-core-9.0.1-myapp.jar
java      24487 24542        root   94r      REG              254,0   105122     268824 /opt/apache-tomcat-7.0.61/webapps/ROOT/WEB-INF/lib/flying-saucer-pdf-9.0.1-myapp.jar
java      24487 24542        root   95r      REG              254,0   105172     268734 /opt/apache-tomcat-7.0.61/webapps/ROOT/WEB-INF/lib/flying-saucer-pdf-itext5-9.0.1-myapp.jar
java      24487 24559        root  mem       REG              254,0   105172     268734 /opt/apache-tomcat-7.0.61/webapps/ROOT/WEB-INF/lib/flying-saucer-pdf-itext5-9.0.1-myapp.jar
java      24487 24559        root  mem       REG              254,0  1106329     268857 /opt/apache-tomcat-7.0.61/webapps/ROOT/WEB-INF/lib/flying-saucer-core-9.0.1-myapp.jar
java      24487 24559        root  mem       REG              254,0   105122     268824 /opt/apache-tomcat-7.0.61/webapps/ROOT/WEB-INF/lib/flying-saucer-pdf-9.0.1-myapp.jar
java      24487 24559        root   93r      REG              254,0  1106329     268857 /opt/apache-tomcat-7.0.61/webapps/ROOT/WEB-INF/lib/flying-saucer-core-9.0.1-myapp.jar
java      24487 24559        root   94r      REG              254,0   105122     268824 /opt/apache-tomcat-7.0.61/webapps/ROOT/WEB-INF/lib/flying-saucer-pdf-9.0.1-myapp.jar
java      24487 24559        root   95r      REG              254,0   105172     268734 /opt/apache-tomcat-7.0.61/webapps/ROOT/WEB-INF/lib/flying-saucer-pdf-itext5-9.0.1-myapp.jar
java      24487 24629        root  mem       REG              254,0   105172     268734 /opt/apache-tomcat-7.0.61/webapps/ROOT/WEB-INF/lib/flying-saucer-pdf-itext5-9.0.1-myapp.jar
java      24487 24629        root  mem       REG              254,0  1106329     268857 /opt/apache-tomcat-7.0.61/webapps/ROOT/WEB-INF/lib/flying-saucer-core-9.0.1-myapp.jar
java      24487 24629        root  mem       REG              254,0   105122     268824 /opt/apache-tomcat-7.0.61/webapps/ROOT/WEB-INF/lib/flying-saucer-pdf-9.0.1-myapp.jar
java      24487 24629        root   93r      REG              254,0  1106329     268857 /opt/apache-tomcat-7.0.61/webapps/ROOT/WEB-INF/lib/flying-saucer-core-9.0.1-myapp.jar
java      24487 24629        root   94r      REG              254,0   105122     268824 /opt/apache-tomcat-7.0.61/webapps/ROOT/WEB-INF/lib/flying-saucer-pdf-9.0.1-myapp.jar
java      24487 24629        root   95r      REG              254,0   105172     268734 /opt/apache-tomcat-7.0.61/webapps/ROOT/WEB-INF/lib/flying-saucer-pdf-itext5-9.0.1-myapp.jar
...snip...

$ lsof | grep flyingsaucer | wc -l
228

So, the jars are loaded, but the app is still unable to find my custom compiled jars which contain the missing method.
What am I doing wrong here? Thanks for any help you can provide.

A:

This looks like a classpath conflict. When starting the Tomcat container try add -verbose to your JVM settings:
export JAVA_OPTS="-verbose"

Then from the output check which JAR the com.lowagie.text.pdf.BaseFont is loaded from and verify it is the correct one. This will help you debug the situation.

