Kristinn Sigurðsson
7c5b2ceb00
Added a data map to Link. Becomes the CrawlURI's data map when Link is
...
promoted to CrawlURI
2013-05-13 15:18:02 +00:00
Kenji Nagahashi
3841d096d5
removed commented-out old code.
2013-05-11 15:44:25 -07:00
Kenji Nagahashi
2825cb736a
script conole: say "line" when linesExecuted==1.
...
generate global variables guide from model.availableGlobalVariables.
2013-05-11 15:44:25 -07:00
Kenji Nagahashi
73f58dc56a
scripting console:
...
fix bug: exception from script execution is not displayed.
remove extra space around exception and rawOutput.
involves a bit more than bug fix. introduced ScriptExecution class for
encapsulating script, execution, and result.
2013-05-11 15:44:25 -07:00
Noah Levitt
e531de0ed3
avoid polluting logs with long stacktraces when a request comes in for a report from an unbuilt job (a fairly common case in archive-it)
2013-04-29 19:02:22 -07:00
Noah Levitt
d5693563f8
fix bug pointed out in https://webarchive.jira.com/browse/ARI-3176?focusedCommentId=35314 - url-agnostic dedupe not reported in host report (and other reports)
2013-04-29 18:11:24 -07:00
Noah Levitt
dafd95eef7
fix for HER-2037 final crawl reports written after crawl is "FINISHED" - call dumpReports() in StatisticsTracker.stop() which (since the HER-1954 changes) runs at the right time, after all url processing has finished but before the FINISHED state is officially reached
2013-04-22 21:23:43 -07:00
Noah Levitt
1713093699
remove stray space
2013-04-22 21:18:35 -07:00
Noah Levitt
81d0a23e50
restore build timestamp
2013-04-09 10:42:23 -07:00
Noah Levitt
3d90506b99
use LinkedHashMap to ensure consistent ordering of xml api
2013-04-09 10:34:02 -07:00
Noah Levitt
08e18fd692
avoid creating unnecessary AlerthThreadGroups
2013-04-05 09:35:23 -07:00
Noah Levitt
475d07e8ca
attempt to address HER-2036 occasional strange memory leak after crawl finishes - have ToePool discard its reference to CrawlController at crawl finish
2013-04-05 09:34:58 -07:00
Noah Levitt
d07bf80e02
add link to engine page on job page
2013-04-03 23:50:23 -07:00
Noah Levitt
5a7cd8a6a8
disable codemirror search which add-on which seems to fight with browser search all the time
2013-04-03 17:16:04 -07:00
Noah Levitt
fd668e82ba
Merge branch 'master' of github.com:internetarchive/heritrix3
2013-04-03 16:39:45 -07:00
Noah Levitt
9f33b3b667
button on engine page to ask java to run garbage collector
2013-04-03 16:39:32 -07:00
Kenji Nagahashi
c98c1d0e29
changed CrawlerLoggerModule.setupLogs() from private to protected so that sub-class can modify logging configuration.
2013-04-02 17:55:03 -07:00
Kenji Nagahashi
4dd46e6af8
quick fix to ClassCastException in ToePool.shortReportLineTo().
...
fixed a typo in Job.ftl.
2013-04-01 23:45:22 -07:00
Noah Levitt
8a40f9ff13
fix misleading logging
2013-04-01 13:04:33 -07:00
Ilya Kreymer
45b205061f
Fix for testCompressedARCStream - if arc has gzip header but doesn't have IA flags, its still compressed
2013-03-31 18:54:12 -07:00
Noah Levitt
6a0ed067c0
consolidate disparate versions of WARCConstants into one version that lives in ia-web-commons
2013-03-29 17:12:32 -07:00
Noah Levitt
1ac1dd9bab
fix typo which removed call to candidate.setFullVia(), should fix failing Precedence3SelfTest
2013-03-29 13:17:58 -07:00
Noah Levitt
dcbbec4bbb
fix for HER-2035 xml representation of job url can be invalid
2013-03-29 12:16:07 -07:00
Noah Levitt
5fdf5713a2
Merge branch 'master' of github.com:internetarchive/heritrix3
2013-03-28 19:10:19 -07:00
Noah Levitt
f988a26e10
move some code into EngineModel constructor and restore reverse chronological ordering of crawl jobs
2013-03-28 19:09:14 -07:00
Noah Levitt
6a2b4a4a76
Merge branch 'master' into freemarker
...
Conflicts:
.classpath
2013-03-28 18:01:16 -07:00
Noah Levitt
8ba69e6706
put .settings in .gitignore after all
2013-03-28 17:53:20 -07:00
Noah Levitt
3665e129f3
put .settings in .gitignore after all
2013-03-28 17:52:13 -07:00
Ilya Kreymer
2df2427e55
Dependency Fixing: Added LaxHttpParser which can be used explicitly when using the lax header parsing functionality.
...
Added HttpParser which just extends LaxHttpParser in the original httpclient package to avoid breaking compatibility with code
which may still reference HttpParser.
Explicit commons code changed to use LaxHttpParser in arc/warc record reading
2013-03-27 18:29:55 -07:00
Noah Levitt
a08d83a370
Merge branch 'master' of github.com:internetarchive/heritrix3
2013-03-27 14:48:32 -07:00
Noah Levitt
69eb124b5d
add test for handling of bad http headers (to make sure we don't break this when we get rid of archive-overlay)
2013-03-27 14:48:21 -07:00
Kenji Nagahashi
a4a3ce60cd
fix failing ContentDigestHistoryTest
2013-03-26 15:06:31 -07:00
Noah Levitt
5d8fe0443d
Merge branch 'master' into HER-2031
2013-03-23 19:16:05 -07:00
Noah Levitt
042ed066b3
add more eclipse stuff to .gitignore
2013-03-20 14:27:34 -07:00
Noah Levitt
8fd361e518
update source/target version to 1.6 since we use 1.6 features, specifically @Override for methods only implementing an interface (not sure why maven wasn't enforcing this); also remove maven-antrun-plugin created timestamp.txt, doesn't seem to be used anywhere
2013-03-20 14:25:22 -07:00
Noah Levitt
c509b6e552
remove more eclipse stuff, people like to manage their own
2013-03-20 13:39:28 -07:00
Noah Levitt
654c106d19
remove eclipse stuff and add to .gitignore (except .settings which has code formatting config)
2013-03-20 12:55:45 -07:00
Noah Levitt
79422099b4
remove old profiling stuff
2013-03-20 12:35:41 -07:00
Noah Levitt
75bb19c4ea
switch .classpath from archive-commons to ia-web-commons
2013-03-19 11:25:13 -07:00
Noah Levitt
9fa198944a
depend on new "ia-web-commons" instead of "archive-commons"
2013-03-18 23:32:05 -07:00
Noah Levitt
b0c6619f80
Merge branch 'master' of github.com:internetarchive/heritrix3
2013-03-18 20:02:03 -07:00
Noah Levitt
0c9e2a1f4d
ContentDigestHistory{Loader,Storer} - ignore uris with empty content body (deduping that is counterproductive)
2013-03-18 20:01:52 -07:00
Noah Levitt
ff36546eda
fix serverless whois url example in comment
2013-03-18 14:03:39 -07:00
Adam Miller
2f4d437734
Beans.ftl
...
Fixed a bug that showed objects with no properties as null instead of displaying the class name
Job.ftl
Fixed a bug that prevented the configuration referenced paths from displaying
Script.ftl
Fixed a bug that rendered extra whitespace before and after the script block
2013-03-15 16:06:06 -07:00
gojomo
c5c54ac138
comment-out debug output
2013-03-12 11:42:47 -07:00
gojomo
7c8295ee82
checkpoint support
2013-03-12 11:42:29 -07:00
gojomo
a844878e7d
Merge branch 'simple_logins' into HER-2031
2013-03-12 01:16:31 -07:00
gojomo
3f8590fe44
when found in CrawlURI, add WARC-Simple-Form-Province-Status header
2013-03-11 20:57:41 -07:00
gojomo
bf9cab8eb2
when POST/submit-data requested, alter request method accordingly
2013-03-11 20:57:09 -07:00
gojomo
bbac7d7320
per (overlay) settings, spawn high-priority POST when appropriate; annotate/add-headers to describe
2013-03-11 20:56:31 -07:00
Kristinn Sigurðsson