Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

ato/BrowserProcessor-fixes

browser-processor-reliability

collectd-socket-stats-writer

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1

#10

#10

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12

#120

#121

#122

#123

#124

#125

#126

#127

#127

#128

#129

#13

#13

#130

#131

#132

#133

#134

#134

#135

#136

#137

#138

#139

#14

#14

#140

#140

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15

#150

#151

#152

#153

#153

#154

#159

#16

#16

#161

#163

#164

#165

#166

#168

#169

#17

#17

#171

#174

#174

#178

#179

#18

#180

#183

#183

#184

#184

#185

#186

#187

#188

#188

#189

#19

#19

#190

#192

#194

#196

#197

#2

#2

#20

#20

#205

#207

#209

#21

#21

#210

#213

#215

#216

#217

#218

#219

#22

#22

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292

#294

#295

#296

#298

#3

#3

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#76

#77

#78

#79

#8

#8

#80

#81

#82

#83

#84

#85

#85

#86

#87

#88

#89

#9

#9

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• 04af55739d We need the bleeding edge httpcomponents (with patches too, but thats not in the pom) Noah Levitt 2012-12-31 10:33:46 -08:00
• a0b62054c9 Fixes to httpcomponents patch to make it build Noah Levitt 2012-12-31 10:27:28 -08:00
• 163871d462 Tweaks and twiddles Noah Levitt 2012-12-31 01:57:03 -08:00
• 25234898ef Somewhat redundant form auth integration test restored, passes Noah Levitt 2012-12-30 22:35:28 -08:00
• fcece02646 Close recorders on failure to avoid exception from retry cycle within httpclient- now all the FetchHTTPTests pass Noah Levitt 2012-12-30 20:45:05 -08:00
• d9311affbc Improve debug logging Noah Levitt 2012-12-30 19:54:04 -08:00
• cb725bb38e Fix socket timeout Noah Levitt 2012-12-30 19:05:55 -08:00
• dac492f2c6 DnsResolver using ServerCache (should add a test for this) Noah Levitt 2012-12-30 01:00:07 -08:00
• d1c41d624f SSL tests pass Noah Levitt 2012-12-30 00:18:44 -08:00
• 17ad459053 Fix bug in httpclient so that midfetch abort works and testShouldFetchBodyRule() passes Noah Levitt 2012-12-29 23:01:28 -08:00
• 7cb05c6961 Make sure wire logger is disabled for testMaxFetchKBSec(), otherwise it can slow things down enough that it fails Noah Levitt 2012-12-29 21:37:46 -08:00
• 2a6a2aa6d4 testHttpProxyAuth() passes Noah Levitt 2012-12-29 21:05:37 -08:00
• 6e8d4efe35 Refactor code building the request into new class FetchHTTPRequest; testHttpProxy() passes Noah Levitt 2012-12-29 20:33:47 -08:00
• 01a9c89088 testHttpBindAddress() passes Noah Levitt 2012-12-29 15:09:46 -08:00
• 959504b7f0 testAcceptCompression() passes Noah Levitt 2012-12-29 14:59:59 -08:00
• c01e50a5a5 Digest auth test passes Noah Levitt 2012-12-29 14:29:47 -08:00
• 70875f1605 Basic auth test passes Noah Levitt 2012-12-29 14:00:36 -08:00
• 34e0ba96ce Get rid of "UmmSession*Buffer* classes and use Session*BufferImpl directly Noah Levitt 2012-12-28 21:23:18 -08:00
• 6fe901d1f8 New approach to overriding SessionInputBuffer seems promising. (httpcomponents.diff is the patch to httpcomponents head that is needed for this h3 commit to work, will keep up to date with each commit) Noah Levitt 2012-12-28 21:06:15 -08:00
• 6300caf367 A little more work on the approach of wrapping the SessionInputBuffer, to remember that overriding BHttpConnectionBase.getSessionInputBuffer() makes it possible (but it doesn't work because we need to get in there and no-op the buffering since we do our own, and it eats up the stream too soon...) Noah Levitt 2012-12-28 18:46:39 -08:00
• 33a500b213 Some work on an approach to overriding SessionInputBuffer that looks like is not the way Noah Levitt 2012-12-28 15:37:58 -08:00
• c9223a9a02 More work on adapting to httpcomponents HEAD with custom patches Noah Levitt 2012-12-22 17:10:38 -08:00
• d3db71e3bb Working on adapting to httpcomponents HEAD with custom patches (hoping to get those incorporated) Noah Levitt 2012-12-21 16:42:30 -08:00
• e26ee96895 * FetchHTTPTests.java testLaxUrlEncoding() - Tests a URL not correctly url-encoded, but that heritrix should let pass through to mimic browser behavior (test fails currently) Noah Levitt 2012-12-20 11:14:16 -08:00
• c7813938e0 Merge branch 'master' into new-fetchhttp-only Noah Levitt 2012-12-20 11:04:42 -08:00
• 543e8820ef Merge branch 'master' into archive-commons-refactor Noah Levitt 2012-12-20 02:44:57 -08:00
• 0de9869c0f Tweaks to new UURIFactory Noah Levitt 2012-12-20 02:42:16 -08:00
• 9168813ed7 Fix bugs in refactoring of UURI/UsableURI Noah Levitt 2012-12-20 02:32:56 -08:00
• 3a65fa25d7 Working on refactoring some stuff into archive-commons Noah Levitt 2012-12-18 14:46:52 -08:00
• 0eac126f7d Special WARC-Profile for uri-agnostic revisit records. See HER-2022, http://tech.groups.yahoo.com/group/archive-crawler/message/7806 Noah Levitt 2012-12-13 14:11:14 -08:00
• 5586d5452d * JerichoExtractorHTMLTest.java avoid redundancy by using extractor built in setUp() makeExtractor() - call extractor.setExtractorJS(new ExtractorJS()) since we got rid of the static ExtractorJS stuff testConditionalComment1() - override to skip the test since it fails with JerichoExtractorHTML Noah Levitt 2012-12-11 16:58:48 -08:00
• 85118c59c4 Tighten up javascript extraction based on real-world analysis (addresses HER-1523) * UriUtils.java new method isVeryLikelyUri() with tighter heuristic than isLikelyUri() * ExtractorJS.java use UriUtils.isVeryLikelyUri(), and change order of operations to do fixup before call to isVeryLikelyUri(), since it doesn't expect strings with javascript escaping and stuff * StringExtractorTestBase.java handle test data with expected value null, meaning no outlinks expected * ExtractorHTMLTest.java avoid redundancy by using the extractor created in ContentExtractorTestBase.setUp() * ExtractorJSTest.java some new tests Noah Levitt 2012-12-11 16:27:26 -08:00
• b1cd3d30a0 * ExtractorHTMLTest.java makeExtractor() - call setExtractorJS(new ExtractorJS()) so testSpeculativeLinkExtraction() passes Noah Levitt 2012-12-10 17:31:55 -08:00
• 256b16249a * ExtractorJS.java refactor so considerStrings() is not static, allowing it to be overridden in subclasses * ExtractorHTML.java, ExtractorJS.java add @Autowired parameter extractorJS used to process inline javascript, instead of call to static ExtractorJS.considerStrings() Noah Levitt 2012-12-10 16:51:42 -08:00
• 6531733972 H3 module for deduplicating with WBM index (with test case). Added EasyMock to project dependency. Kenji Nagahashi 2012-12-10 14:37:18 -08:00
• b2a90ddc15 * UriUtils.java speculativeFixup() - improve detection of scheme-less intended-absolute-URIs Noah Levitt 2012-12-10 14:10:34 -08:00
• d3dc153c5f * ModuleTestBase.java runTest() - convenience logging of test failures Noah Levitt 2012-12-10 13:56:45 -08:00
• e44cf9ca9b A little cleanup of FetchHTTPTest refactoring Noah Levitt 2012-12-04 18:43:15 -08:00
• 5354960f00 Refactor FetchHTTPTest to use a TestSuite so that starting and shutting down the test http servers happen only once at start and finish respectively Noah Levitt 2012-12-04 17:25:09 -08:00
• 2c0dbed399 .classpath include junit as a regular dependency not managed by eclipse, so source jar can be attached Noah Levitt 2012-12-04 17:20:39 -08:00
• e30cb83540 * FetchHTTPTest.java testLaxUrlEncoding() - Tests a URL not correctly url-encoded, but that heritrix lets pass through to mimic browser behavior. Noah Levitt 2012-12-04 16:25:09 -08:00
• b981fd7423 * ArchiveReader.java add "implements Closeable" since it already has the close() method Noah Levitt 2012-12-03 17:09:22 -08:00
• 4496c5d755 * FetchHTTPTest.java move to modules/src/test testFormAuth() - remove, there is FormAuthSelfTest.java, and this was the test that would fail in modules/ testHttpBindAddress() - incorporate ed65493db0 - alternate testing of multiple local binds more likely to work more places Noah Levitt 2012-11-30 17:57:36 -08:00
• dc7fea3252 * .classpath add missing libs Noah Levitt 2012-11-30 17:56:53 -08:00
• a20824aea7 Merge branch 'master' into new-fetchhttp-only Noah Levitt 2012-11-30 17:04:28 -08:00
• 92b302f164 * ExtractorMultipleRegexTest.java avoid "constant string too long" compile error Noah Levitt 2012-11-16 19:08:18 -08:00
• 94df568b4f * ExtractorMultipleRegex.java remove temporary performance testing code Noah Levitt 2012-11-16 18:34:28 -08:00
• aa98ec3327 * ExtractorMultipleRegex.java keep cache of groovy Template objects, since they are expensive to create Noah Levitt 2012-11-16 18:30:41 -08:00
• eeee01a52d * ExtractorMultipleRegex.java javadocs Noah Levitt 2012-11-16 16:42:43 -08:00
• 89dfef468c * ExtractorMultipleRegexTest.java test passes now; extractor gets what it can, which is most of the scroll down urls Noah Levitt 2012-11-15 19:18:59 -08:00
• dfd7a8a7d3 * ExtractorMultipleRegexTest.java turns out that __adt parameter can be found near the json blob - most, but not all, expected links are now found Noah Levitt 2012-11-15 18:58:11 -08:00
• 22ea489c57 * ExtractorMultipleRegex.java remove the "fooIndex" thing from available bindings, since it's kinda hacky and turned out not to be needed for our use case Noah Levitt 2012-11-15 18:56:54 -08:00
• 4af58f38e5 * CrawlURI.java outLinks, outCandidates - use LinkedHashSet to ensure predictable order (any reason not to do that?) Noah Levitt 2012-11-15 18:55:37 -08:00
• 75280a13a8 * ExtractorMultipleRegexTest.java working on making test work - first two scroll-down urls are extracted successfully, others fail Noah Levitt 2012-11-15 17:27:41 -08:00
• 13e1f9d64d * ExtractorMultipleRegex.java fix omission from last commit, uriRegex Noah Levitt 2012-11-15 14:01:01 -08:00
• 6ec4b5a973 * ExtractorMultipleRegex.java more refactoring to avoid redundant operations Noah Levitt 2012-11-15 13:55:11 -08:00
• 43dbcbfe81 * ExtractorMultipleRegex.java refactor for readability Noah Levitt 2012-11-15 13:35:16 -08:00
• 3f44a4bc66 Merge branch 'master' into MoreImpliedExtractor Noah Levitt 2012-11-15 12:35:17 -08:00
• 8395e372e4 * ScriptedProcessor.java getEngine() - fix logic to respect isolateThreads setting Noah Levitt 2012-11-15 12:33:48 -08:00
• ba5bca807e * ExtractorMultipleRegex.java use groovy templating facility Noah Levitt 2012-11-15 12:32:42 -08:00
• 6e2d2129fe * ExtractorMultipleRegexTest.java use real use case actual content, actual urls from facebook for testing * ExtractorMultipleRegex.java some comments, little twiddles Noah Levitt 2012-11-15 11:16:57 -08:00
• 1e2d86dc63 * ContentExtractorTestBase.java createRecorder(String,String) - new method for creating test Recorder and specifying charset createRecorder(String) - deprecate (use default charset as before) Noah Levitt 2012-11-15 11:14:25 -08:00
• 1d994493e2 more progress on ExtractorMultipleRegex Noah Levitt 2012-11-14 19:58:43 -08:00
• 1998337645 tmp commit Travis Wellman 2012-11-14 17:58:44 -08:00
• ed65493db0 alternate testing of multiple local binds more likely to work more places gojomo 2012-10-01 16:38:17 -07:00
• b4baea0a13 make maven requested version match eclipse project classpath gojomo 2012-10-01 16:37:31 -07:00
• 37e6ab802a Merge branch 'master' of github.com:internetarchive/heritrix3 gojomo 2012-10-01 13:41:12 -07:00
• 4470ba512e Count url-agnostic content digest duplicates as duplicates in statistics * CrawledBytesHistotable.java accumulate() - count urls with warcRevisit:uriAgnosticDigest annotation as duplicates * ContentDigestHistoryTest.java testWarcDedupe() - check for presence of warcRevisit:uriAgnosticDigest annotation Noah Levitt 2012-09-27 16:56:14 -07:00
• a1d135edbc * WARCWriterProcessor.java include warcFileOffset in CrawlURI extraInfo (optionally logged in crawl log if logExtraInfo is enabled) Noah Levitt 2012-09-27 16:51:50 -07:00
• e2a1e3bb53 * ContentDigestHistoryTest.java testWarcDedupe() - open the written with with WARCReader, check that each record is as expected Noah Levitt 2012-09-27 14:34:20 -07:00
• a1a5fc7379 * WARCWriterProcessor.java writeRevisitUriAgnosticDigest() - put angle brackets around WARC-Refers-To uri value, since warc record ids appear everywhere with those brackets Noah Levitt 2012-09-27 14:20:00 -07:00
• 7a92e1ff7c * FetchHTTPTest.java shut down test servers in tearDown() (server startup/shutdown happens on every test, inefficient but it doesn't take all that long anyway) * ContentDigestHistoryTest.java shut down test servers in finally block Noah Levitt 2012-09-26 19:29:11 -07:00
• e6fbe45774 Another test for ContentDigestHistory * ContentDigestHistoryTest.java new test testWarcDedupe() fetches two different urls with same content, writes warc records, checks results * FetchHTTPTest.java, WARCWriterProcessorTest.java make test module code public and static for use in ContentDigestHistoryTest.java Noah Levitt 2012-09-26 18:54:48 -07:00
• f9703fb5ef * engine/pom.xml remove incorrect, unnecessary dependency Noah Levitt 2012-09-26 14:55:03 -07:00
• b9cbbbd248 Bunch of unit tests for FetchHTTP * FetchHTTPTest.java a bunch of new tests * FetchHTTP.java make sslTrustLevel work correctly (bug found by test; even with fix, only respects value at startup, no overrides) * modules/pom.xml, .classpath dependencies for testing * engine/pom.xml update jetty to 6.1.26 since we require that for the new tests Noah Levitt 2012-09-26 14:41:49 -07:00
• f0b4cfeeac Merge branch 'master' into new-fetchhttp-only Noah Levitt 2012-09-26 12:59:41 -07:00
• ca5802f7ba separated hq from local hertirix3 repo, fixed up layout Kenji Nagahashi 2012-09-26 12:31:55 -07:00
• 3612396ff3 Unit test for content digest history * ContentDigestHistoryTest.java test basics * BdbContentDigestHistory.java code formatting tweak Noah Levitt 2012-09-26 12:03:45 -07:00
• 190e2ff0f3 Fix unit test * WARCWriterTest.java writeRandomHTTPRecord() - fix method, had transposed some lines of code Noah Levitt 2012-09-26 10:58:01 -07:00
• 2e1d81fa80 Fix build * WARCConstants.java, WARCWriter.java remove imports of WARCWriterProcessor from the modules/ area (was only used in javadoc) Noah Levitt 2012-09-26 08:58:22 -07:00
• 50c8a09b63 * BdbContentDigestHistory.java class javadoc Noah Levitt 2012-09-25 16:04:44 -07:00
• 819371f6eb * BdbContentDigestHistory.java store() - avoid clobbering good content digest history with empty one Noah Levitt 2012-09-25 15:28:19 -07:00
• 34252c4949 * WARCWriterProcessor.java writeHttpRecords() - avoid inadvertently creating content digest history map Noah Levitt 2012-09-25 15:26:56 -07:00
• fe1df5883f Some tweaks on maintaining content digest history * CrawlURI.java utility method hasContentDigestHistory() * BdbContentDigestHistory.java load() - make sure to call curi.getContentDigestHistory() in all cases so the value is initialized and WARCWriterProcessor knows it should put the info in there * WARCWriterProcessor.java updateMetadataAfterWrite() - update curi.getContentDigestHistory() only if curi.hasContentDigestHistory() for efficiency, like old uri-based fetch history; update the count after writing a revisit record Noah Levitt 2012-09-25 14:35:26 -07:00
• 25a9155298 New parent class AbstractContentDigestHistory for implementations to extend * AbstractContentDigestHistory.java abstract methods load(), store(); non-abstract persistKeyFor(); javadocs * BdbContentDigestHistory.java extend AbstractContentDigestHistory * ContentDigestHistoryLoader.java, ContentDigestHistoryStorer.java use AbstractContentDigestHistory Noah Levitt 2012-09-25 13:38:25 -07:00
• b8e14ba1df Rename ContentDigestHistory -> BdbContentDigestHistory, since it's bdb-specific Noah Levitt 2012-09-25 13:12:59 -07:00
• 7965a27b1d Fix infinite loop bug on server returning no response * RecordingSocketInputBuffer.java readLine(CharArrayBuffer) - return -1 (not 0) on end of stream * RecordingHttpClient.java setHttpRequestRetryHandler() - never retry (may want to reevaluate later, old fetcher did retry sometimes) * FetchHTTPTest.java testNoResponse() Noah Levitt 2012-09-13 19:10:11 -07:00
• dd7e0c5bd2 * profile-crawler-beans.cxml replace defunct commons-httpclient BdbCookieStorage with new httpcomponents BdbCookieStore Noah Levitt 2012-09-13 17:58:47 -07:00
• 77559bf81c **/pom.xml bump version number because of api change Noah Levitt 2012-09-13 17:47:26 -07:00
• ef78e584c4 * WARCWriterProcessor.java writeRevisitUriAgnosticDigest() - write several separate warc header fields for each value instead of one json blob * WARCConstants.java new header field names Noah Levitt 2012-09-13 17:21:29 -07:00
• a8aae91167 * WARCConstants.java remove rfc6648-deprecated "X-" prefix to "WARC-Reference-Location", and document the field * WARCWriterProcessor.java writeRevisitUriAgnosticDigest() - add some comments Noah Levitt 2012-09-11 18:21:20 -07:00
• 1048536a25 * WARCWriterProcessor.java writeRevisitUriAgnosticDigest() - fix construction of X-WARC-Reference-Location json value Noah Levitt 2012-09-11 17:21:34 -07:00
• 38aefe4d16 Write revisit records for URI-agnostic content digest duplicates. * WARCConstants.java new header keys "WARC-Refers-To" and "X-WARC-Reference-Location" * CrawlURI.java getContentDigestHistory() - return HashMap instead of Map so clone() can be called on the return value * WARCWriterProcessor.java writeHttpRecords(), writeRevisitUriAgnosticDigest() - write revisit record for URI-agnostic content digest duplicate if enabled Noah Levitt 2012-09-11 17:12:24 -07:00
• a04e76f00f Load, store content digest history * WARCRecordInfo.java new fields warcFilename, warcFileOffset * WARCWriter.java temporarily remember warc records written * CrawlURI.java convenience method getContentDigestHistory() * ContentDigestHistory.java start up properly; load into and store from curi.getContentDigestHistory() * RecrawlAttributeConstants.java bunch of hash keys for curi.getContentDigestHistory() * WARCWriterProcessor.java save info in curi.getContentDigestHistory() Noah Levitt 2012-09-11 10:10:49 -07:00
• 2407fc5dd0 Beginning work on deduplication by content digest (irrespective of url) * ContentDigestHistory.java, ContentDigestHistoryLoader.java, ContentDigestHistoryStorer.java initial checkin Noah Levitt 2012-09-10 18:18:12 -07:00
• c4faa444a2 * ANVLRecord.java remove unneeded(?) serialVersionUID Noah Levitt 2012-09-10 18:14:15 -07:00
• a414f522f6 Continuing to work on cleaning up WARC writer code: simply creation of WARC record headers. * ANVLRecord.java extend LinkedList instead of ArrayList (it was inane to guess the number of records everywhere, when it was never much more than ~10) * ANVLRecords.java remove (seemed to be an incomplete duplicate of ANVLRecord) * WARCRecordInfo.java addExtraHeader() - convenience method * Arc2Warc.java, WARCWriter.java, WARCWriter.java, WARCWriterProcessor.java use WARCRecordInfo.addExtraHeader() where appropriate, and use no-argument ANVLRecord constructor instead of deprecated ANVLRecord(int) Noah Levitt 2012-09-10 18:03:25 -07:00
• 360ac00000 Continuing to work on cleaning up WARC writer code. * WARCRecordInfo.java get rid of constructor with arguments, add setUrl() and setType() * WARCWriter.java get rid of writeRecord() taking a million arguments * Arc2Warc.java, WARCWriter.java, WARCWriterTest.java, WARCWriterProcessor.java, WARCWriterProcessorTest.java construct WARCRecordInfo object and call WARCWriter.writeRecord(recordInfo) instead of million argument method Noah Levitt 2012-09-10 17:41:43 -07:00
• dd5e51fd9d Working on cleaning up WARC writer code. * WARCRecordInfo.java new class to hold warc record info to write, so we don't have to pass around all these long lists of variables * WARCConstants.java new enum WARCRecordType replaces bunch of string constants * WARCWriter.java, WARCWriterProcessor.java, WARCWriterTest.java use WARCRecordInfo, WARCRecordType Noah Levitt 2012-09-10 15:43:37 -07:00
• 395220f5bf get rid of archive-overlays-commons-httpclient Noah Levitt 2012-09-08 01:57:14 -07:00