Alex Osborne
ab8a5efea2
Format HTTP request lines in API guide
2018-07-04 18:24:08 +09:00
Alex Osborne
e35eb9e850
Move conventions and REST section to the end of the document
...
Let's dive right into the API. Chances are anyone who wants to use
the REST API already knows what it is. :-)
2018-07-04 17:53:05 +09:00
Alex Osborne
57f465706e
Convert html tables to rst tables
2018-07-04 17:34:19 +09:00
Alex Osborne
90db2e33d5
Add skeleton sphinx docs for readthedocs
2018-07-04 16:40:51 +09:00
Kristinn Sigurðsson
05811705ed
Point at GitHub wiki for latest release info
2018-05-29 09:42:24 +00:00
Noah Levitt
7308911807
Merge pull request #205 from adam-miller/ParallelQueueRandomAssignment
...
Add parameter to allow even distribution for parallel queues.
2018-05-10 15:18:11 -07:00
Adam Miller
bee53de8b1
Add parameter to allow even distribution for parallel queues.
2018-05-10 21:34:02 +00:00
Barbara Miller
40bfe9f79f
Merge pull request #197 from nlevitt/ari-5630
...
catch exceptions scoping outlinks to stop them from derailing process…
2018-01-24 16:31:33 -08:00
Noah Levitt
95759149ae
catch exceptions scoping outlinks to stop them from derailing processing of the parent url
2018-01-16 17:12:24 -08:00
Noah Levitt
5a0326eba1
Merge pull request #196 from kngenie/fix-test-failures-with-bdb
...
fix for test failures in a workspace on NFS-mounted filesystem
2017-12-12 14:20:55 -08:00
Kenji Nagahashi
a7b7c6cf5a
fix for test failures in a workspace on NFS-mounted filesystem
...
ContentDigestHistoryTest does not close BdbModule. It results in failure to delete bdb directory in following tests.
Added tearDown() method that closes BdbModule.
2017-12-08 17:15:03 -08:00
Noah Levitt
76bf98a4f0
Merge pull request #194 from internetarchive/formsMaxSize
...
limit max size of form input
2017-11-16 11:24:20 -08:00
Barbara Miller
998edf3479
max size for extracted form elements
2017-11-16 08:48:06 -08:00
Noah Levitt
deda35eedd
Merge pull request #192 from internetarchive/robotstxt-size-limit
...
Enforce robots.txt character limit per char not per line
2017-11-01 17:20:30 -07:00
Alex Osborne
d6d7f1b162
Fix ArrayIndexOutOfBoundsException when all lines are blank
2017-11-01 10:05:26 +09:00
Alex Osborne
68ceceedd2
Use simpler read 500k chars and split method suggested by @nlevitt
...
While we temporarily use a little more memory this version is a lot
less codes. It also allows us to do away with the BufferedReader.
2017-11-01 09:29:28 +09:00
Alex Osborne
1fae77c64d
Enforce robots.txt character limit per char not per line
...
Since we were only enforcing the size limit after reading each
line a single very long line could cause us to to run out of
memory.
This change enforces the character limit on each read character
not just on line boundaries.
We also correct the count of processed characters in the warning
message, which was not counting newline and linefeed characters.
2017-10-31 18:41:00 +09:00
Barbara Miller
f0d1c0422b
Merge pull request #187 from nlevitt/robots-allow-failed-fetch
...
treat a failed fetch (e.g. socket timeout) of robots.txt the same way…
2017-10-13 11:23:54 -07:00
Noah Levitt
dd88191bac
Merge pull request #190 from ukwa/fix-ci-and-add-java-8
...
Allow JavaDNS to be disabled as part of resolving outstanding build and test issues
2017-10-13 11:09:36 -07:00
Andrew Jackson
aee83dfe26
Allow DNS settings to be overridden via sheets.
2017-10-13 10:29:15 +01:00
Andrew Jackson
e7ce1f56b4
Ensure disabling JavaDNS always fails over to the InetAddress approach.
2017-10-12 23:33:54 +01:00
Andrew Jackson
d178c689de
Add some comments to try to avoid misuse.
2017-10-12 22:13:00 +01:00
Andrew Jackson
d85d2a3fe8
Move OneLineSimpleLayout into commons so all can use it.
2017-10-12 22:10:58 +01:00
Andrew Jackson
b0594ac0f4
Allow JavaDNS to be disabled and use that to avoid issues in StatisticsSelfTest.
2017-10-12 21:29:34 +01:00
Andrew Jackson
f7a5376fbe
Merge branch 'master' into fix-ci-and-add-java-8
2017-10-12 09:31:47 +01:00
Andrew Jackson
09ee95f37b
Allow results with or without DNS lookup of localhost.
2017-10-11 23:30:17 +01:00
Noah Levitt
ae8a0ef95f
update webarchive-commons dependency to incorporate fix to warc size calculation, and uncomment test that exercises the fix
2017-10-11 14:54:37 -07:00
Noah Levitt
4e96d0c94f
Merge pull request #189 from adam-miller/master
...
WARCLimitEnforcer.java - Add support for multiple warc writers.
2017-10-11 14:46:58 -07:00
Adam Miller
b702bf3d86
WARCLimitEnforcer.java - Add support for multiple warc writers.
2017-10-11 21:06:35 +00:00
Andrew Jackson
46fa54da0c
Set an appropriate Content Size when using non-DNS resolves.
2017-10-11 17:08:15 +01:00
Andrew Jackson
a5381440fd
Use build folder for temp files as tmp gets wiped on Travis CI sometimes.
2017-10-09 22:13:53 +01:00
Andrew Jackson
401e27fc64
Adapt test to cope with variation in how localhost gets resolved.
2017-10-09 21:50:05 +01:00
Jaimie Murdock
a0a783c750
oraclejdk7 EOL'd in May 2015, not supported in Travis-CI Trusty image
2017-10-09 16:42:32 -04:00
Andrew Jackson
1de6f7d2a0
Update Travis CI file to suitable Java 7 + 8 variants.
2017-10-09 09:54:16 +01:00
Noah Levitt
84debcfe4d
treat a failed fetch (e.g. socket timeout) of robots.txt the same way as http errors: full allow
2017-10-06 12:00:31 -07:00
Barbara Miller
0ec055c211
Merge pull request #186 from nlevitt/trough-logging-tweak
...
reduce batch size to 400 and avoid ridiculously long log lines
2017-10-04 16:32:11 -07:00
Noah Levitt
d01f2c3a30
fix copy&paste mistake
2017-10-04 16:29:22 -07:00
Noah Levitt
8fea6c4783
reduce batch size to 400 and avoid ridiculously long log lines
2017-10-04 15:35:16 -07:00
jkafader
65004ea748
Merge pull request #185 from internetarchive/trough-esc-sql
...
escape strings in sql posted to trough
2017-09-21 12:19:35 -07:00
Noah Levitt
7fc1c2f25f
escape strings in sql posted to trough
2017-09-21 12:16:25 -07:00
jkafader
4c438d3852
Merge pull request #180 from internetarchive/aitfive-1176
...
trough feed
2017-09-05 17:37:36 -07:00
Noah Levitt
1c92be2d74
realized this stuff needs to be thread safe
2017-06-27 16:01:15 -07:00
Noah Levitt
66ead24762
missed another comma
2017-06-27 15:50:33 -07:00
Noah Levitt
698a02a5b7
missed a comma
2017-06-27 15:45:55 -07:00
Noah Levitt
2f29949b88
include http response payload when logging errors from trough
2017-06-26 17:37:39 -07:00
Noah Levitt
3892c17b56
ughhh appease java 8 javadoc rules
2017-06-15 17:29:22 -07:00
Noah Levitt
a1a7232530
update for new data model queued_url->uncrawled_url and some new fields
2017-06-14 18:31:39 -07:00
Noah Levitt
b670c2f59a
missed the maven tarball assembly file
2017-06-08 11:39:08 -07:00
Noah Levitt
a1f59da000
oops, String.join is a java 8 feature
2017-06-08 11:35:50 -07:00
Noah Levitt
87a15bddc8
Merge pull request #168 from internetarchive/gh-167
...
fix for race-condition when first using the WARC writers https://gith…
2017-06-07 18:15:19 -07:00
Alex Osborne