<commit_before><!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html><head><meta content="text/html; charset=ISO-8859-1" http-equiv="content-type"><title>Notes on TDS Security Audit</title></head>
<body><h1>Notes on TDS Security Audit</h1><hr style="width: 100%; height: 2px;"><span style="font-weight: bold;"></span><ul><li><a href="#Issues">Issues</a></li><li><a href="#Summary">Summary</a></li><li><a href="#To_Do">To Do</a></li><li><a href="#OWASP_References">OWASP References</a></li><li><a href="#Tools_for_encoding_and_decoding">Tools for encoding and decoding</a></li><li><a href="#Unicode_Notes">Unicode notes</a></li><li><a href="#TDS_Notes">TDS notes</a></li></ul><hr style="width: 100%; height: 2px;"><h2><a name="Issues"></a>Issues</h2><ul><li>validate input [to prevent lots of things]</li><li>output filtering/encoding [to prevent XSS]</li><li>log filtering/encoding [to prevent Log Forging]</li><li>don't allow any stackTrace information out in responses (or in output stream?) [to prevent System Information Leak]</li><ul><li>setup error pages (500.html, etc) so servlet and IO exceptions don't get exposed to user</li><li>stop using Exception.pringStackTrace() (??? how does this get into response? only in catalina.out?)</li></ul><li>Check
CDM validator: netCDF var names and att names and values should
be&nbsp;filtered/encoded before output into XML or HTML (or logged?)</li><li>Check Catalog Services especially when servicing a remote catalog</li></ul><hr style="width: 100%; height: 2px;"><h2><a name="Summary"></a>Summary</h2><p>Input Validation</p><ul><li>Check expected type: boolean, long, int, etc</li><li>String (??) - maybe:</li><ul><li>should whitespace be allowed</li><li>should CR/LF be allowed</li><li>should any control characters be allowed</li><li>should quotes be allowed</li></ul><li>TDS specific:</li><ul><li>path information:</li><ul><li>spring StringUtils.trimAllWhitespace()? except probably want to allow spaces</li><li>write a trimAllControlCharacters() similar to StringUtils.trimAllWhitespace()</li><li>Do no validation but be sure to encode for logging and any XML/HTML output???</li></ul></ul></ul><p>Output filter/encode:</p><ul><li>HTML output</li><ul><li>encode HTML special characters as character entities. See</li><ul><li>org.apache.commons.lang.StringEscapeUtils.escapeHtml(), and</li><li>org.springframework.web.util.HtmlUtils.htmlEscape()</li></ul><li>???</li></ul><li>XML output</li><ul><li>encode XML special characters as character entities. See:</li><ul><li>org.apache.commons.lang.StringEscapeUtils.escapeXml() - only five special characters? see javadocs</li></ul></ul><li>Other output</li><ul><li>depends on what it is I guess? nc, GeoTIFF, </li></ul></ul><p>Log messages:</p><ul><li>validate/filter - don't allow/filter out&nbsp;CR and LF ??</li><li>encode all non-alphanumeric characters - see java.net.URLEncoder</li></ul><p>Check for validation/filtering/encoding when user input is used in:</p><ul><li>HttpServletResponse.getOutputStream()
- if user input is sent over the output stream [?not
for&nbsp;HttpServletResponse.getWriter()??]</li><li>HttpServletResponse.addHeader() - if user input is used in header value (or name!?)</li><li>HttpServletResponse.setStatus() - if includes message and user input is put into message</li><li>HttpServletResponse.sendError() - if includes message and user input is put into message&nbsp;</li></ul>Just don't do it:<ul><li>Exception.printStackTrace()</li></ul><hr style="width: 100%; height: 2px;"><h2><a name="To_Do"></a>To Do</h2>Questions for <NAME>:<p>In our code:</p><ul><li>Write validators, filters, and encoders for various cases:</li><ul><li>TDS path validation (what are acceptable characters for file system path/name? See conclusion of <a href="#Unicode_Notes">Unicode notes section</a> below)</li><li>encode HTML content (Is HTML character reference encoding enough? What about Unicode control characters?)</li><li>encode XML content (Is HTML character reference encoding enough? What about Unicode control characters?)</li><li>other output encoding (Depends on output format.)</li><li>encode&nbsp;log
messages (Could just use java.net.URLEncoder on any untrusted content?
But don't really need the "+" in place of spaces.)</li></ul><li>Write class that implements javax.servlet.Filter and filter all user input</li><ul><li>Possibly write class that extends javax.servlet.http.HttpServletRequestWrapper</li></ul><li>Find all places in code where validator, filter, encoders are needed and use them.</li><li>Stop&nbsp;System Information Leakage (see OWASP stuff below) in error messages and such</li><ul><li>Stop using Exception.printStackTrace() in favor of logging</li><li>Stop using System.out.println() in favor of logging</li><li>Setup error pages (500.html, etc) so Servlet and IO exceptions don't get exposed to user</li></ul></ul><hr style="width: 100%; height: 2px;"><h2><a name="OWASP_References"></a>OWASP References</h2>Some OWASP&nbsp;<a href="http://www.owasp.org/index.php/Category:Attack">Attack</a> and <a href="http://www.owasp.org/index.php/Category:Vulnerability">Vulnerability</a> pages:<br><ul><li>XSS- Cross-site Scripting (<a href="http://www.owasp.org/index.php/Cross-site_Scripting_%28XSS%29">OWASP XSS page</a>)</li><li>CSRF - Cross-Site Request Forgery (<a href="http://www.owasp.org/index.php/Cross-Site_Request_Forgery">OWASP CSRF page</a>)</li><li>Log Forging (<a href="http://www.owasp.org/index.php/Log_Forging">OWASP</a>)</li><li>Log injection (<a href="http://www.owasp.org/index.php/Log_injection">OWASP</a>)</li><li>HTTP Response Splitting (<a href="http://www.owasp.org/index.php/HTTP_Response_Splitting">OWASP</a>) - involves unvalidated input (inluding CR ["%0D" or "\r"] and LF ["%0A" and "\n"]) making its way into HTTP response headers</li><li>System Information Leak (<a href="http://www.owasp.org/index.php/HTTP_Response_Splitting">OWASP</a>)</li></ul><hr style="width: 100%; height: 2px;"><h2><a name="Tools_for_encoding_and_decoding"></a>Tools for encoding and decoding</h2><h3>Servlet API</h3><p>A few things to be aware of:</p><ul><li>req.getPathInfo():String - the response is URL decoded</li><li>req.getParameter():String
- the response is URL decoded (at least if params are part of query
string [GET], not sure about if in body [POST])</li><li>req.getQueryString():String - the response is&nbsp;<strong>NOT</strong> URL decoded</li><li>req.getRequestURI():String - the response is&nbsp;<strong>NOT</strong> URL decoded [API does not specify if decoded or not; others here are specified. When using Tomcat 6.0.18, it was not decoded.]</li><li>req.getRequestURL():StringBuffer - the response is&nbsp;<strong>NOT</strong> URL decoded </li></ul><h3>java.net.URLEncoder and URLDecoder</h3><p>Encode and decode in and out of "<code>application/x-www-form-urlencoded</code>" MIME format. Rules summary:</p><ul><li>[a-z], [A-Z], and [0-9] &nbsp;remain the same</li><li>"<code>.</code>",
     "<code>-</code>", "<code>*</code>", and
     "<code>_</code>" remain the same</li><li>the space character (' ') is encoded as a plus sign ('+') and vice versa for decoding</li><li>all
other characters are converted to bytes in the named character encoding
and each byte is encoded as "%hh". [Note: decoding goes from "%hh" to
bytes and then characters with&nbsp;illegal character handling
(either&nbsp;throw IllegalArgumentException or leave characters as is)
left up to the implementation.]</li></ul><p>Note: from the HTML spec [Section "<a href="http://www.w3.org/TR/html4/interact/forms.html#h-17.13.4.1">Form content type</a>"]: "[l]ine breaks are represented as "CR LF" pairs (i.e.,
<samp>`%0D%0A'</samp>)".</p><p>Methods:</p><ul><li>URLEncoder.encode(String s, String charEncoding):String</li><li>URLDecoder.decode(String s, String charEncoding):String</li></ul><p>Javadocs: <a href="http://java.sun.com/javase/6/docs/api/java/net/URLEncoder.html">URLEncoder</a> and <a href="http://java.sun.com/javase/6/docs/api/java/net/URLDecoder.html">URLDecoder</a></p><h3>org.apache.commons.lang</h3><p>Jar file size is ~250Kb</p><p>StringEscapeUtils.escape*() - where * can be Html, Xml, Java, JavaScript, Sql, Csv</p><p>Does not support URL forms escaping.</p><p>Javadocs: <a href="http://commons.apache.org/lang/api-release/org/apache/commons/lang/StringEscapeUtils.html">StringEscapeUtils</a></p><h3>org.springframework</h3><p>Jar file size is ~2.8Mb</p><h4>org.springframework.web.util.HtmlUtils</h4><p>HTML 4.0 compliant escaping of strings (handles complete character set defined in HTML 4.01)</p><p><a href="http://static.springframework.org/spring/docs/2.5.x/api/org/springframework/web/util/HtmlUtils.html">JavaDocs</a> say:</p><p style="margin-left: 40px;">"For a comprehensive set of String escaping utilities,
 consider Jakarta Commons Lang and its StringEscapeUtils class.
 We are not using that class here to avoid a runtime dependency
 on Commons Lang just for HTML escaping. Furthermore, Spring's
 HTML escaping is more flexible and 100% HTML 4.0 compliant.
"</p><p>Methods:</p><ul><li>htmlEscape(String): String - convert special characters to entity reference (e.g., '&lt;' to "&amp;lt;")</li><li>htmlEscapeDecimal(String): String - same but to decimal encoding ("&amp;#<var>Decimal</var>;")</li><li>htmlEscapeHex(String):String - same but to hex encoding ("&amp;#x<var>Hex</var>;")</li><li>htmlUnescape(String):String - unescapes all of the above</li></ul><h4>org.springframework.web.util.UrlPathHelper</h4><p> Helper class for URL path matching. Also support for URL paths in RequestDispatcher includes.</p><p>Could replace a lot in thredds.servlet.ServletUtils</p><p><a href="http://static.springframework.org/spring/docs/2.5.x/api/org/springframework/web/util/UrlPathHelper.html">Javadoc</a></p><h4>org.springframework.util.StringUtils</h4><p>Lots of String utilities. <a href="http://static.springframework.org/spring/docs/2.5.x/api/org/springframework/util/StringUtils.html">Javadoc</a>
say:&nbsp;&nbsp;"[m]ainly for internal use within the [Spring]
framework; consider Jakarta's Commons Lang for a more comprehensive
suite of String utilities".</p><ul><li>cleanPath(String):String - "Normalize the path by suppressing sequences like "path/.." and
 inner simple dots."</li><li>applyRelativePath( String path, String relativePath):String</li><li>containsWhitespace(String):boolean -&nbsp;</li><li>trimWhitespace() - leading and trailing</li><li>trimAllWhitespace() - leading, trailing, and between characters</li><li>pathEquals(String,String):boolean - compare after normalizing with cleanPath().</li></ul><hr style="width: 100%; height: 2px;"><h2><a name="Unicode_Notes"></a>Unicode Notes</h2><h3>Allowing Unicode characters in file paths/names</h3><p> Review the Unicode spec&nbsp;to determine which Unicode
characters, if they are in a string that will be used as a file
path/name, should:</p><ol><li> cause the string to be deemed invalid,</li><li>be filtered or encoded.</li></ol><p>[Note: Started looking at Unicode as I was considering how to filter/validate/encode TDS path
information.]</p><p><a href="http://www.unicode.org/">Unicode</a> Specification <a href="http://www.unicode.org/versions/Unicode5.1.0/">version 5.1.0</a></p><p>Section 4.5 "General Category - Normative" describes the set of general categories:</p><ul><li> L - letter (Lu - upper, Ll - lower, Lt - title, Lm - modifier, Lo -other)</li><li>M - mark (Mn - non-spacing, Mc - spacing combining, Me - enclosing)</li><li>N - number (Nd - decimal digit, Nl - letter, No - other)</li><li>P - punctuation (Pc - connector, Pd - dash, Ps - open, Pe - close, Pi - initial quote, Pf - final quote, Po - other)</li><li>S - symbol (Sm - math, Sc - currency, Sk - modifier, So - other)</li><li>Z - separator (Zs - space, Zl - line, Zp - paragraph)</li><li>C - other (Cc - control, Cf - format, Cs - surrogate [reserved for UTF-16], Co - private use, Cn - not assigned)</li></ul><p>It
also mentions that "[a characters general category] is ... used to
support common APIs such as isDigit()". However, it continues with:</p><p style="margin-left: 40px;">"Common
functions such as isLetter()and isUppercase()do not extend well to the
larger and more complex repertoire of Unicode. While it is possible to
naively extend these functions to Unicode using the General Category
and other properties, they will not work for the entire range of
Unicode characters and range of tasks for which people use them. For
more appropriate approaches, see Unicode Standard Annex #31,
&#8220;Identifier and Pattern Syntax&#8221;; Unicode Standard Annex #29, &#8220;Text
Boundaries&#8221;; Section 5.18, Case Mappings; and Section 4.10, Letters, Alphabetic, and Ideographic." </p><p>Table
2-3 "Types of Code Points" (in section 2.4 "Code Points and
Characters") contains better descriptions for some of the general
categories.&nbsp;And even better, <a href="http://www.fileformat.info/">FileFormat.Info</a> has some <a href="http://www.fileformat.info/info/unicode/index.htm">good information on Unicode</a> including a <a href="http://www.fileformat.info/info/unicode/category/index.htm">list of categories</a> each linking to <strong>a description that includes a list of the characters in that category</strong>. (Also a <a href="http://www.fileformat.info/info/unicode/block/index.htm">list of the various blocks</a>.)</p><p>What
about Unicode normalization? What if a person types in a string in a
non-normalized form, we normalized in one way, the filesystem
normalizes (or not) in another way? Is it then hard for them to</p><ol><li> access the file from the command-line if it was created by TDS; or</li><li>access the file from the TDS if it was created from the command-line?</li></ol><p>Conclusion:</p><ol><li>Not sure about Unicode normalization (???).</li><li>For
strings that are to be used as a file path/name,
any characters in the&nbsp;Unicode general categories listed here
should not be allowed (they should&nbsp;probably make the given string
invalid rather than being encoded or filtered as they probably indicate
the user is up to no-good (???)):</li><ul><li>Zl - Seperator, Line</li><li>Zp - Seperator, Paragraph</li><li>Cc - Other, Control</li><li>Cs - Other, Surrogate</li><li>Co - Other, Private Use</li><li>Cn - Other, Not Assigned</li></ul></ol><p></p><hr style="width: 100%; height: 2px;"><h2><a name="TDS_Notes"></a>TDS Notes</h2><ul><li>[DONE] General filter on request path [URL pattern: "/*"]</li><li>[DONE] General filter on request query string [URL pattern: "/*"]</li><li>Specific filters:</li><ul><li>[DONE] CatalogService parameters [URL pattern: "/catalog.html", "/catalogServices", "/catalog/*", "/catalogGen/*"]:</li><ul><li>catalog - single value, URI</li><li>dataset - single value, ID</li><li>cmd - single value, alphanumeric string constrained to "show", "subset", "validate", "convert"</li><li>debug - single value, boolean</li></ul><li>[DONE] CatalogGen parameters [URL pattern: "/catalogGen/*"]:</li><ul><li>taskName - single value, ID</li><li>fileName - single value, File</li><li>resultFileName - single value, File</li><li>period - single value, decimal number</li><li>delay - single value, decimal number</li></ul><li>[STARTED] CoordSysValidator parameters [URL pattern: "/cdmValidate"]:</li><ul><li>GET only:</li><ul><li>URL</li><li>xml</li></ul><li>POST: some file upload stuff</li></ul></ul><li>Services that can operate on remote datasets and catalogs by default don't support operation on remote resources</li><ul><li>WCS:</li><ul><li>[DONE] only allow if threddsConfig.xml WCS.allowRemote is set to true</li><li>[???]Any filtering on the remote dataset needed for secure use of this feature?</li></ul><li>CdmValidator</li><ul><li>[] only allow if threddsConfig.xml CdmValidatorService.allowRemote is set to true</li><li>[???] what filtering is needed if remote datasets are allowed</li></ul><li>CatalogService</li><ul><li>[] only allow remote catalog if threddsConfig.xml CatalogService.allowRemote is set to true</li><li>[] what filtering should be done on remote catalog if allowed</li></ul></ul><li>Handle&nbsp;when ServletException or IOException is thrown by doGet() et al</li><ul><li>[X] Check on default: 500 response with exception stack trace shown</li><li>Decide on appropriate course:</li><ul><li>[X] Replace default 500 page with own 500.html that doesn't show stack trace</li><li>[] Write filter that wraps 500 response in a 400 response</li><li>[] Catch any such exceptions or errors and setStatus() etc</li></ul><li>[X] Implement: replace default 404 and 500 pages</li></ul><li>[] Block CoordSysValidator POST</li><li>[] Block other POST and PUT stuff</li><li>[] TEST, TEST, TEST</li></ul><br><h3>Test URLs</h3><ul><li>http://localhost:8080/thredds/cataloggen/%3Cscript%3Ealert(%27Whoops%27)%3C/script%3E</li><li>http://localhost:8080/thredds/catalogServices?%3Cscript%3Ealert(%27Whoops%27)%3C/script%3E</li></ul><hr style="width: 100%; height: 2px;"></body></html><commit_msg>More notes on TDS 3.16 web security review.<commit_after><!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html><head><meta content="text/html; charset=ISO-8859-1" http-equiv="content-type"><title>Notes on TDS Security Audit</title></head>
<body><h1>Notes on TDS Security Audit</h1><hr style="width: 100%; height: 2px;"><span style="font-weight: bold;"></span><ul><li><a href="#Issues">Issues</a></li><li><a href="#Summary">Summary</a></li><li><a href="#To_Do">To Do</a></li><li><a href="#OWASP_References">OWASP References</a></li><li><a href="#Tools_for_encoding_and_decoding">Tools for encoding and decoding</a></li><li><a href="#Unicode_Notes">Unicode notes</a></li><li><a href="#TDS_Notes">TDS notes</a></li></ul><hr style="width: 100%; height: 2px;"><h2><a name="Issues"></a>Issues</h2><ul><li>validate input [to prevent lots of things]</li><li>output filtering/encoding [to prevent XSS]</li><li>log filtering/encoding [to prevent Log Forging]</li><li>don't allow any stackTrace information out in responses (or in output stream?) [to prevent System Information Leak]</li><ul><li>setup error pages (500.html, etc) so servlet and IO exceptions don't get exposed to user</li><li>stop using Exception.pringStackTrace() (??? how does this get into response? only in catalina.out?)</li></ul><li>Check
CDM validator: netCDF var names and att names and values should
be&nbsp;filtered/encoded before output into XML or HTML (or logged?)</li><li>Check Catalog Services especially when servicing a remote catalog</li></ul><hr style="width: 100%; height: 2px;"><h2><a name="Summary"></a>Summary</h2><p>Input Validation</p><ul><li>Check expected type: boolean, long, int, etc</li><li>String (??) - maybe:</li><ul><li>should whitespace be allowed</li><li>should CR/LF be allowed</li><li>should any control characters be allowed</li><li>should quotes be allowed</li></ul><li>TDS specific:</li><ul><li>path information:</li><ul><li>spring StringUtils.trimAllWhitespace()? except probably want to allow spaces</li><li>write a trimAllControlCharacters() similar to StringUtils.trimAllWhitespace()</li><li>Do no validation but be sure to encode for logging and any XML/HTML output???</li></ul></ul></ul><p>Output filter/encode:</p><ul><li>HTML output</li><ul><li>encode HTML special characters as character entities. See</li><ul><li>org.apache.commons.lang.StringEscapeUtils.escapeHtml(), and</li><li>org.springframework.web.util.HtmlUtils.htmlEscape()</li></ul><li>???</li></ul><li>XML output</li><ul><li>encode XML special characters as character entities. See:</li><ul><li>org.apache.commons.lang.StringEscapeUtils.escapeXml() - only five special characters? see javadocs</li></ul></ul><li>Other output</li><ul><li>depends on what it is I guess? nc, GeoTIFF, </li></ul></ul><p>Log messages:</p><ul><li>validate/filter - don't allow/filter out&nbsp;CR and LF ??</li><li>encode all non-alphanumeric characters - see java.net.URLEncoder</li></ul><p>Check for validation/filtering/encoding when user input is used in:</p><ul><li>HttpServletResponse.getOutputStream()
- if user input is sent over the output stream [?not
for&nbsp;HttpServletResponse.getWriter()??]</li><li>HttpServletResponse.addHeader() - if user input is used in header value (or name!?)</li><li>HttpServletResponse.setStatus() - if includes message and user input is put into message</li><li>HttpServletResponse.sendError() - if includes message and user input is put into message&nbsp;</li></ul>Just don't do it:<ul><li>Exception.printStackTrace()</li></ul><hr style="width: 100%; height: 2px;"><h2><a name="To_Do"></a>To Do</h2>Questions for <NAME>:<p>In our code:</p><ul><li>Write validators, filters, and encoders for various cases:</li><ul><li>TDS path validation (what are acceptable characters for file system path/name? See conclusion of <a href="#Unicode_Notes">Unicode notes section</a> below)</li><li>encode HTML content (Is HTML character reference encoding enough? What about Unicode control characters?)</li><li>encode XML content (Is HTML character reference encoding enough? What about Unicode control characters?)</li><li>other output encoding (Depends on output format.)</li><li>encode&nbsp;log
messages (Could just use java.net.URLEncoder on any untrusted content?
But don't really need the "+" in place of spaces.)</li></ul><li>Write class that implements javax.servlet.Filter and filter all user input</li><ul><li>Possibly write class that extends javax.servlet.http.HttpServletRequestWrapper</li></ul><li>Find all places in code where validator, filter, encoders are needed and use them.</li><li>Stop&nbsp;System Information Leakage (see OWASP stuff below) in error messages and such</li><ul><li>Stop using Exception.printStackTrace() in favor of logging</li><li>Stop using System.out.println() in favor of logging</li><li>Setup error pages (500.html, etc) so Servlet and IO exceptions don't get exposed to user</li></ul></ul><hr style="width: 100%; height: 2px;"><h2><a name="OWASP_References"></a>OWASP References</h2>Some OWASP&nbsp;<a href="http://www.owasp.org/index.php/Category:Attack">Attack</a> and <a href="http://www.owasp.org/index.php/Category:Vulnerability">Vulnerability</a> pages:<br><ul><li>XSS- Cross-site Scripting (<a href="http://www.owasp.org/index.php/Cross-site_Scripting_%28XSS%29">OWASP XSS page</a>)</li><li>CSRF - Cross-Site Request Forgery (<a href="http://www.owasp.org/index.php/Cross-Site_Request_Forgery">OWASP CSRF page</a>)</li><li>Log Forging (<a href="http://www.owasp.org/index.php/Log_Forging">OWASP</a>)</li><li>Log injection (<a href="http://www.owasp.org/index.php/Log_injection">OWASP</a>)</li><li>HTTP Response Splitting (<a href="http://www.owasp.org/index.php/HTTP_Response_Splitting">OWASP</a>) - involves unvalidated input (inluding CR ["%0D" or "\r"] and LF ["%0A" and "\n"]) making its way into HTTP response headers</li><li>System Information Leak (<a href="http://www.owasp.org/index.php/HTTP_Response_Splitting">OWASP</a>)</li></ul><hr style="width: 100%; height: 2px;"><h2><a name="Tools_for_encoding_and_decoding"></a>Tools for encoding and decoding</h2><h3>Servlet API</h3><p>A few things to be aware of:</p><ul><li>req.getPathInfo():String - the response is URL decoded</li><li>req.getParameter():String
- the response is URL decoded (at least if params are part of query
string [GET], not sure about if in body [POST])</li><li>req.getQueryString():String - the response is&nbsp;<strong>NOT</strong> URL decoded</li><li>req.getRequestURI():String - the response is&nbsp;<strong>NOT</strong> URL decoded [API does not specify if decoded or not; others here are specified. When using Tomcat 6.0.18, it was not decoded.]</li><li>req.getRequestURL():StringBuffer - the response is&nbsp;<strong>NOT</strong> URL decoded </li></ul><h3>java.net.URLEncoder and URLDecoder</h3><p>Encode and decode in and out of "<code>application/x-www-form-urlencoded</code>" MIME format. Rules summary:</p><ul><li>[a-z], [A-Z], and [0-9] &nbsp;remain the same</li><li>"<code>.</code>",
     "<code>-</code>", "<code>*</code>", and
     "<code>_</code>" remain the same</li><li>the space character (' ') is encoded as a plus sign ('+') and vice versa for decoding</li><li>all
other characters are converted to bytes in the named character encoding
and each byte is encoded as "%hh". [Note: decoding goes from "%hh" to
bytes and then characters with&nbsp;illegal character handling
(either&nbsp;throw IllegalArgumentException or leave characters as is)
left up to the implementation.]</li></ul><p>Note: from the HTML spec [Section "<a href="http://www.w3.org/TR/html4/interact/forms.html#h-17.13.4.1">Form content type</a>"]: "[l]ine breaks are represented as "CR LF" pairs (i.e.,
<samp>`%0D%0A'</samp>)".</p><p>Methods:</p><ul><li>URLEncoder.encode(String s, String charEncoding):String</li><li>URLDecoder.decode(String s, String charEncoding):String</li></ul><p>Javadocs: <a href="http://java.sun.com/javase/6/docs/api/java/net/URLEncoder.html">URLEncoder</a> and <a href="http://java.sun.com/javase/6/docs/api/java/net/URLDecoder.html">URLDecoder</a></p><h3>org.apache.commons.lang</h3><p>Jar file size is ~250Kb</p><p>StringEscapeUtils.escape*() - where * can be Html, Xml, Java, JavaScript, Sql, Csv</p><p>Does not support URL forms escaping.</p><p>Javadocs: <a href="http://commons.apache.org/lang/api-release/org/apache/commons/lang/StringEscapeUtils.html">StringEscapeUtils</a></p><h3>org.springframework</h3><p>Jar file size is ~2.8Mb</p><h4>org.springframework.web.util.HtmlUtils</h4><p>HTML 4.0 compliant escaping of strings (handles complete character set defined in HTML 4.01)</p><p><a href="http://static.springframework.org/spring/docs/2.5.x/api/org/springframework/web/util/HtmlUtils.html">JavaDocs</a> say:</p><p style="margin-left: 40px;">"For a comprehensive set of String escaping utilities,
 consider Jakarta Commons Lang and its StringEscapeUtils class.
 We are not using that class here to avoid a runtime dependency
 on Commons Lang just for HTML escaping. Furthermore, Spring's
 HTML escaping is more flexible and 100% HTML 4.0 compliant.
"</p><p>Methods:</p><ul><li>htmlEscape(String): String - convert special characters to entity reference (e.g., '&lt;' to "&amp;lt;")</li><li>htmlEscapeDecimal(String): String - same but to decimal encoding ("&amp;#<var>Decimal</var>;")</li><li>htmlEscapeHex(String):String - same but to hex encoding ("&amp;#x<var>Hex</var>;")</li><li>htmlUnescape(String):String - unescapes all of the above</li></ul><h4>org.springframework.web.util.UrlPathHelper</h4><p> Helper class for URL path matching. Also support for URL paths in RequestDispatcher includes.</p><p>Could replace a lot in thredds.servlet.ServletUtils</p><p><a href="http://static.springframework.org/spring/docs/2.5.x/api/org/springframework/web/util/UrlPathHelper.html">Javadoc</a></p><h4>org.springframework.util.StringUtils</h4><p>Lots of String utilities. <a href="http://static.springframework.org/spring/docs/2.5.x/api/org/springframework/util/StringUtils.html">Javadoc</a>
say:&nbsp;&nbsp;"[m]ainly for internal use within the [Spring]
framework; consider Jakarta's Commons Lang for a more comprehensive
suite of String utilities".</p><ul><li>cleanPath(String):String - "Normalize the path by suppressing sequences like "path/.." and
 inner simple dots."</li><li>applyRelativePath( String path, String relativePath):String</li><li>containsWhitespace(String):boolean -&nbsp;</li><li>trimWhitespace() - leading and trailing</li><li>trimAllWhitespace() - leading, trailing, and between characters</li><li>pathEquals(String,String):boolean - compare after normalizing with cleanPath().</li></ul><hr style="width: 100%; height: 2px;"><h2><a name="Unicode_Notes"></a>Unicode Notes</h2><h3>Allowing Unicode characters in file paths/names</h3><p> Review the Unicode spec&nbsp;to determine which Unicode
characters, if they are in a string that will be used as a file
path/name, should:</p><ol><li> cause the string to be deemed invalid,</li><li>be filtered or encoded.</li></ol><p>[Note: Started looking at Unicode as I was considering how to filter/validate/encode TDS path
information.]</p><p><a href="http://www.unicode.org/">Unicode</a> Specification <a href="http://www.unicode.org/versions/Unicode5.1.0/">version 5.1.0</a></p><p>Section 4.5 "General Category - Normative" describes the set of general categories:</p><ul><li> L - letter (Lu - upper, Ll - lower, Lt - title, Lm - modifier, Lo -other)</li><li>M - mark (Mn - non-spacing, Mc - spacing combining, Me - enclosing)</li><li>N - number (Nd - decimal digit, Nl - letter, No - other)</li><li>P - punctuation (Pc - connector, Pd - dash, Ps - open, Pe - close, Pi - initial quote, Pf - final quote, Po - other)</li><li>S - symbol (Sm - math, Sc - currency, Sk - modifier, So - other)</li><li>Z - separator (Zs - space, Zl - line, Zp - paragraph)</li><li>C - other (Cc - control, Cf - format, Cs - surrogate [reserved for UTF-16], Co - private use, Cn - not assigned)</li></ul><p>It
also mentions that "[a characters general category] is ... used to
support common APIs such as isDigit()". However, it continues with:</p><p style="margin-left: 40px;">"Common
functions such as isLetter()and isUppercase()do not extend well to the
larger and more complex repertoire of Unicode. While it is possible to
naively extend these functions to Unicode using the General Category
and other properties, they will not work for the entire range of
Unicode characters and range of tasks for which people use them. For
more appropriate approaches, see Unicode Standard Annex #31,
&#8220;Identifier and Pattern Syntax&#8221;; Unicode Standard Annex #29, &#8220;Text
Boundaries&#8221;; Section 5.18, Case Mappings; and Section 4.10, Letters, Alphabetic, and Ideographic." </p><p>Table
2-3 "Types of Code Points" (in section 2.4 "Code Points and
Characters") contains better descriptions for some of the general
categories.&nbsp;And even better, <a href="http://www.fileformat.info/">FileFormat.Info</a> has some <a href="http://www.fileformat.info/info/unicode/index.htm">good information on Unicode</a> including a <a href="http://www.fileformat.info/info/unicode/category/index.htm">list of categories</a> each linking to <strong>a description that includes a list of the characters in that category</strong>. (Also a <a href="http://www.fileformat.info/info/unicode/block/index.htm">list of the various blocks</a>.)</p><p>What
about Unicode normalization? What if a person types in a string in a
non-normalized form, we normalized in one way, the filesystem
normalizes (or not) in another way? Is it then hard for them to</p><ol><li> access the file from the command-line if it was created by TDS; or</li><li>access the file from the TDS if it was created from the command-line?</li></ol><p>Conclusion:</p><ol><li>Not sure about Unicode normalization (???).</li><li>For
strings that are to be used as a file path/name,
any characters in the&nbsp;Unicode general categories listed here
should not be allowed (they should&nbsp;probably make the given string
invalid rather than being encoded or filtered as they probably indicate
the user is up to no-good (???)):</li><ul><li>Zl - Seperator, Line</li><li>Zp - Seperator, Paragraph</li><li>Cc - Other, Control</li><li>Cs - Other, Surrogate</li><li>Co - Other, Private Use</li><li>Cn - Other, Not Assigned</li></ul></ol><p></p><hr style="width: 100%; height: 2px;"><h2><a name="TDS_Notes"></a>TDS Notes</h2><ul><li>[DONE] General filter on request path [URL pattern: "/*"]</li><li>[DONE] General filter on request query string [URL pattern: "/*"]</li><li>Specific filters:</li><ul><li>[DONE] CatalogService parameters [URL pattern: "/catalog.html", "/catalogServices", "/catalog/*", "/catalogGen/*"]:</li><ul><li>catalog - single value, URI</li><li>dataset - single value, ID</li><li>cmd - single value, alphanumeric string constrained to "show", "subset", "validate", "convert"</li><li>debug - single value, boolean</li></ul><li>[DONE] CatalogGen parameters [URL pattern: "/catalogGen/*"]:</li><ul><li>taskName - single value, ID</li><li>fileName - single value, File</li><li>resultFileName - single value, File</li><li>period - single value, decimal number</li><li>delay - single value, decimal number</li></ul><li>[STARTED] CoordSysValidator parameters [URL pattern: "/cdmValidate"]:</li><ul><li>GET only:</li><ul><li>URL</li><li>xml</li></ul><li>POST: some file upload stuff</li></ul></ul><li>Services that can operate on remote datasets and catalogs by default don't support operation on remote resources</li><ul><li>WCS:</li><ul><li>[DONE] only allow if threddsConfig.xml WCS.allowRemote is set to true</li><li>[???]Any filtering on the remote dataset needed for secure use of this feature?</li></ul><li>CdmValidator</li><ul><li>[] only allow if threddsConfig.xml CdmValidatorService.allowRemote is set to true</li><li>[XXX] Only works remotely, already by default not allowed (threddsConfig.xml CdmValidatorService.allow=false)</li><li>[???] what filtering is needed if remote datasets are allowed</li></ul><li>CatalogService</li><ul><li>[DONE] only allow remote catalog if threddsConfig.xml CatalogServices.allowRemote is set to true</li><li>[] what filtering should be done on remote catalog if allowed</li></ul></ul><li>Handle&nbsp;when ServletException or IOException is thrown by doGet() et al</li><ul><li>[X] Check on default: 500 response with exception stack trace shown</li><li>Decide on appropriate course:</li><ul><li>[X] Replace default 500 page with own 500.html that doesn't show stack trace</li><li>[] Write filter that wraps 500 response in a 400 response</li><li>[] Catch any such exceptions or errors and setStatus() etc</li></ul><li>[X] Implement: replace default 404 and 500 pages</li></ul><li>[] Block CoordSysValidator POST</li><ul><li>[XXX] Does not work by default (threddsConfig.xml CdmValidatorService.allow=false)</li></ul><li>[DONE] Block other POST and PUT stuff</li><li>[] TEST, TEST, TEST</li></ul><h3>Changed Files</h3><p>Starting with SVN repository version&nbsp;6045, the following files have been changed/added on the 3.16 branch:</p><ul><li>Utilities</li><ul><li>ServletUtil</li><li>HtmlWriter</li><li>LogUtil</li><li>tds/src/main/java/thredds/util/UserInputValidateEncodeUtils (and TestUserInputValidateEncodeUtils)</li></ul><li>Servlets</li><ul><li>ThreddsDefaultServlet</li><li>CatalogServicesServlet</li><li>NetcdfServlet</li><li>WCSServlet</li><li>StationObsServlet</li><li>RadarServer</li><li>FmrcInventoryServlet</li><li>GridServlet</li><li>CatalogGenServlet</li><li>DqcServlet</li><li>DqcServletRedirect</li><li>CoordSysValidatorServlet</li><li>AnnotateServlet [though never used]</li></ul><li>Filters</li><ul><li>RequestPathFilter</li><li>RequestQueryFilter</li><li>CatalogGenUserInputFilter</li><li>CatalogServiceUserInputFilter</li><li>CoordSysValidatorUserInputFilter</li></ul><li>Other</li><ul><li>web-thredds.xml</li><li>threddsConfig.xml</li><li>CatGenTimerTask</li><li>InvDatasetImpl</li><li>tds/src/main/webapp/resources/500.html</li><li>tds/src/main/webapp/resources/404.html</li><li>build.xml</li><li>commons-lang-2.4.jar</li><li>ExampleCatalogWriter</li><li>[UI] CatalogSearcher</li><li>[UI] HtmlBrowser</li></ul></ul><h3>Test URLs</h3><ul><li>[bad] http://localhost:8080/thredds/cataloggen/%3Cscript%3Ealert(%27Whoops%27)%3C/script%3E</li><li>[bad] http://localhost:8080/thredds/catalogServices?%3Cscript%3Ealert(%27Whoops%27)%3C/script%3E</li><li>[OK] /thredds/radarServer/terminal/level3/IDD?var=TR0&amp;stn=MIA&amp;time_end=present&amp;time_duration=PT24H</li></ul><hr style="width: 100%; height: 2px;"></body></html>
{'fraction_non_alphanumeric': 0.17044721715968036, 'fraction_numerical': 0.01306603112294967, 'mean_word_length': 6.134626925385077, 'pattern_counts': {'":': 0, '<': 1286, '<?xml version=': 0, '>': 1286, 'https://': 0, 'lorem ipsum': 0, 'www.': 34, 'xml': 13}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1209547', 'n_tokens_mistral': 11579, 'n_tokens_neox': 10930, 'n_words': 3154}