Eric Leclerc 5f431de215 Update to ExtractorHTML.java for cond. comments ...

Lot's of websites are using "Downlevel Revealed" conditional comments with a twist to have a page with valid html. this is an update to the RegEx to allow such a case. ex:

    <!--[if expression]><!-->
      HTML
    <!--<![endif]-->

- reference: https://css-tricks.com/downlevel-hidden-downlevel-revealed/
- reference: http://www.sitepoint.com/web-foundations/internet-explorer-conditional-comments/
- example site where this type of tag can be found: https://www.canada.ca/index.html

2016-02-17 14:49:50 -05:00

..

main

Update to ExtractorHTML.java for cond. comments

2016-02-17 14:49:50 -05:00

test

Simplify logic for urls with spaces, and make it better follow the pattern of UriUtils.isVeryLikelyUri(). Involves a subtle adjustment to the regex LIKELY_RELATIVE_URI_PATTERN to ensure the 2nd capturing group always gets the file extension. Also add a couple of tests of strings with spaces and file extensions that are not known good extensions.

2016-02-09 15:02:11 -08:00