Commit Graph

Select branches

Hide Pull Requests

ARI-5439

adam/add-crawl-log-logging-to-extractoryoutubedl

adam/add_configurable_extractor_js_script_attribute_rules

adam/ait-add-crawl-log-logging-to-extractoryoutubedl

adam/fix_configurable_js_extractor_strict_overlay

adam/fix_crawl_end_time_checkpoint_resumption

adam/knowledgable_extractorjs_extends_configurable_extractorjs

adam/restore-has-prerequisite-behavior

adam/strip_invalid_xml_chars_from_rest_api_output

adds-bdbmodule-max-file-size-option

adds-subqueue-support-forced-queue-assignment

adds-trough-dedup-performance-rework

adds-trough-local-dedup-write-cache

ait-qa

ait-ydl-updates

ato/BrowserProcessor-fixes

browser-processor-reliability

collectd-socket-stats-writer

fix-yt-dlp-typo-too

fixes-WARCWriterChainProcessor-checkpoint-stats

fixes-extractor-multiple-regex-matcher-recycle

fixes-leaky-file-handles

fixes-prerequisite-status

fixes-trough-client-synchronization

fixes-trough-crawl-log-feed-synchronization

less_youtubedl

master

master-ait-contrib

master-it

meta-content-name

noplaylist-ydl

restlet_npe

revert-367-fixes-trough-crawl-log-feed-synchronization

ydl-no-cache

yt-dl-format

yt-dl-format-medium-ish

yt-dlp

yt-dlp-20230706

yt-dlp-4-h3

yt-dlp-for-h3

#1

#1

#10

#10

#100

#101

#102

#103

#104

#105

#106

#107

#108

#109

#11

#11

#110

#111

#112

#113

#114

#115

#116

#117

#118

#119

#12

#12

#120

#121

#122

#123

#124

#125

#126

#127

#127

#128

#129

#13

#13

#130

#131

#132

#133

#134

#134

#135

#136

#137

#138

#139

#14

#14

#140

#140

#141

#142

#143

#144

#145

#146

#147

#148

#149

#15

#15

#150

#151

#152

#153

#153

#154

#159

#16

#16

#161

#163

#164

#165

#166

#168

#169

#17

#17

#171

#174

#174

#178

#179

#18

#180

#183

#183

#184

#184

#185

#186

#187

#188

#188

#189

#19

#19

#190

#192

#194

#196

#197

#2

#2

#20

#20

#205

#207

#209

#21

#21

#210

#213

#215

#216

#217

#218

#219

#22

#22

#220

#222

#223

#224

#226

#227

#23

#230

#231

#235

#238

#24

#240

#241

#242

#243

#244

#246

#248

#249

#25

#251

#253

#255

#256

#257

#26

#260

#262

#269

#27

#270

#271

#272

#273

#276

#28

#280

#281

#283

#285

#286

#287

#288

#289

#29

#290

#292

#292

#294

#295

#296

#298

#3

#3

#30

#300

#301

#302

#303

#304

#306

#308

#31

#310

#312

#313

#314

#315

#317

#32

#320

#323

#325

#326

#327

#328

#329

#33

#330

#331

#333

#334

#335

#336

#34

#341

#342

#345

#346

#347

#348

#349

#35

#352

#354

#355

#356

#357

#359

#36

#362

#364

#365

#366

#367

#368

#369

#37

#370

#373

#374

#375

#377

#379

#38

#381

#382

#383

#384

#385

#386

#388

#388

#389

#39

#390

#391

#393

#394

#395

#397

#398

#399

#4

#4

#40

#400

#401

#402

#403

#404

#405

#406

#407

#408

#409

#41

#410

#411

#412

#414

#415

#416

#418

#42

#421

#423

#424

#43

#431

#433

#434

#435

#44

#441

#442

#444

#445

#448

#45

#450

#452

#454

#456

#457

#459

#46

#460

#461

#465

#468

#47

#470

#475

#476

#478

#48

#480

#483

#484

#486

#488

#489

#49

#490

#491

#492

#493

#5

#5

#50

#503

#51

#52

#53

#534

#535

#536

#537

#538

#539

#54

#542

#543

#544

#549

#55

#55

#550

#551

#552

#554

#555

#56

#565

#566

#567

#57

#572

#573

#574

#575

#576

#577

#578

#579

#58

#580

#581

#582

#583

#586

#589

#59

#590

#591

#592

#593

#594

#595

#597

#598

#599

#6

#6

#60

#600

#601

#602

#603

#605

#606

#607

#608

#609

#61

#610

#611

#612

#614

#615

#616

#617

#618

#619

#62

#621

#622

#623

#624

#625

#626

#627

#628

#629

#63

#630

#631

#632

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#649

#65

#650

#651

#652

#653

#654

#655

#656

#657

#659

#66

#660

#661

#663

#664

#665

#667

#668

#669

#67

#670

#671

#672

#673

#674

#675

#676

#677

#678

#679

#68

#680

#681

#682

#683

#684

#685

#686

#69

#691

#692

#693

#694

#695

#696

#697

#7

#7

#70

#700

#701

#703

#704

#705

#706

#707

#708

#709

#71

#710

#712

#713

#719

#72

#721

#722

#723

#725

#728

#73

#730

#731

#732

#733

#734

#735

#736

#737

#738

#739

#74

#740

#741

#742

#744

#745

#746

#75

#750

#751

#755

#756

#757

#758

#759

#76

#760

#761

#761

#77

#78

#79

#8

#8

#80

#81

#82

#83

#84

#85

#85

#86

#87

#88

#89

#9

#9

#90

#91

#92

#93

#94

#95

#96

#97

#98

#98

#99

3.0.0

3.1.1

3.10.0

3.10.1

3.10.2

3.11.0

3.12.0

3.13.0

3.14.0

3.14.1

3.15.0

3.16.0

3.2.0

3.4.0-20190205

3.4.0-20190207

3.4.0-20190418

3.4.0-20200304

3.4.0-20200518

3.4.0-20210527

3.4.0-20210617

3.4.0-20210618

3.4.0-20210621

3.4.0-20210803

3.4.0-20210923

3.4.0-20220727

3.4.0-20240909

3.5.0

3.6.0

3.7.0

3.8.0

3.9.0

Mono Color

• 0508abb9ad new contrib subproject for extra generally useful modules not part of the main distribution Noah Levitt 2013-06-05 13:12:36 -07:00
• ffd248f780 HER-2040 ExtractorJS shouldExtract when content-type is application/json Noah Levitt 2013-06-03 11:23:16 -07:00
• 3a54600a79 ADD unit test for XmlMarshaller Kenji Nagahashi 2013-05-29 12:03:49 -07:00
• c53b821c85 FIX: 73f58dc56a broke XML variant of scripting console response, and will produce incompatible structure (extra nesting with scriptExec element). moved ScriptExecution code to ScriptModel, which is now a bean, not Map. XmlMarshaller is extended to support simple JavaBeans to XML marshalling, borrowing annotation classes from javax.xml. Kenji Nagahashi 2013-05-28 13:28:37 -07:00
• 7f2aacba2c BeanLookupBindings: check appCtx != null explicitly, rather than using assert. Kenji Nagahashi 2013-05-28 13:20:23 -07:00
• 73888819dc fix bug where scripting console no longer retains your script when you hit execute Noah Levitt 2013-05-15 14:23:53 -07:00
• 858523db94 HER-2039: Initial work. Eliminated Link class and dealt with resultant errors. Not tested any further than that. Kristinn Sigurðsson 2013-05-15 13:36:01 +00:00
• d4c5bd6e98 avoid npe by setting default <input> type "text" when unspecified Noah Levitt 2013-05-14 18:17:09 -07:00
• d972731958 ExtractorHTMLForms.java - handle a corner case extracting attribute values Noah Levitt 2013-05-14 16:23:41 -07:00
• 7bd010e3c7 HTMLForm.java - default html <input> type is text, so consider <input> with no type as candidateUsernameInput Noah Levitt 2013-05-13 22:53:37 -07:00
• bea5c6f267 followup on HER-2031 ExtractorHTMLForms.java - improve regexes looking for form and input attributes, mainly to avoid maxing cpu for long periods of time for certain input Noah Levitt 2013-05-13 22:37:02 -07:00
• 5b4d8c7f0d use intended File.separator instead of File.pathSeparator Noah Levitt 2013-05-13 22:06:09 -07:00
• ec824e9724 avoid need for getRawData() method by introducing Link.hasData() Noah Levitt 2013-05-13 17:43:45 -07:00
• 7c5b2ceb00 Added a data map to Link. Becomes the CrawlURI's data map when Link is promoted to CrawlURI Kristinn Sigurðsson 2013-05-13 15:18:02 +00:00
• 3841d096d5 removed commented-out old code. Kenji Nagahashi 2013-05-11 15:42:53 -07:00
• 2825cb736a script conole: say "line" when linesExecuted==1. generate global variables guide from model.availableGlobalVariables. Kenji Nagahashi 2013-05-11 15:14:15 -07:00
• 73f58dc56a scripting console: fix bug: exception from script execution is not displayed. remove extra space around exception and rawOutput. involves a bit more than bug fix. introduced ScriptExecution class for encapsulating script, execution, and result. Kenji Nagahashi 2013-05-10 17:54:40 -07:00
• 03c2075eb1 latest httpclient is 4.3-beta2-SNAPSHOT Noah Levitt 2013-05-08 10:21:26 -07:00
• e531de0ed3 avoid polluting logs with long stacktraces when a request comes in for a report from an unbuilt job (a fairly common case in archive-it) Noah Levitt 2013-04-29 19:02:22 -07:00
• d5693563f8 fix bug pointed out in https://webarchive.jira.com/browse/ARI-3176?focusedCommentId=35314 - url-agnostic dedupe not reported in host report (and other reports) Noah Levitt 2013-04-29 18:11:24 -07:00
• dafd95eef7 fix for HER-2037 final crawl reports written after crawl is "FINISHED" - call dumpReports() in StatisticsTracker.stop() which (since the HER-1954 changes) runs at the right time, after all url processing has finished but before the FINISHED state is officially reached Noah Levitt 2013-04-22 21:23:43 -07:00
• 1713093699 remove stray space Noah Levitt 2013-04-22 21:18:35 -07:00
• 81d0a23e50 restore build timestamp Noah Levitt 2013-04-09 10:42:23 -07:00
• 3d90506b99 use LinkedHashMap to ensure consistent ordering of xml api Noah Levitt 2013-04-09 10:34:02 -07:00
• 08e18fd692 avoid creating unnecessary AlerthThreadGroups Noah Levitt 2013-04-05 09:35:23 -07:00
• 475d07e8ca attempt to address HER-2036 occasional strange memory leak after crawl finishes - have ToePool discard its reference to CrawlController at crawl finish Noah Levitt 2013-04-05 09:34:58 -07:00
• d07bf80e02 add link to engine page on job page Noah Levitt 2013-04-03 23:50:23 -07:00
• 5a7cd8a6a8 disable codemirror search which add-on which seems to fight with browser search all the time Noah Levitt 2013-04-03 17:16:04 -07:00
• fd668e82ba Merge branch 'master' of github.com:internetarchive/heritrix3 Noah Levitt 2013-04-03 16:39:45 -07:00
• 9f33b3b667 button on engine page to ask java to run garbage collector Noah Levitt 2013-04-03 16:39:32 -07:00
• c98c1d0e29 changed CrawlerLoggerModule.setupLogs() from private to protected so that sub-class can modify logging configuration. Kenji Nagahashi 2013-04-02 17:55:03 -07:00
• 4dd46e6af8 quick fix to ClassCastException in ToePool.shortReportLineTo(). fixed a typo in Job.ftl. Kenji Nagahashi 2013-04-01 23:45:22 -07:00
• 8a40f9ff13 fix misleading logging Noah Levitt 2013-04-01 13:04:33 -07:00
• 45b205061f Fix for testCompressedARCStream - if arc has gzip header but doesn't have IA flags, its still compressed Ilya Kreymer 2013-03-31 18:54:12 -07:00
• 6a0ed067c0 consolidate disparate versions of WARCConstants into one version that lives in ia-web-commons Noah Levitt 2013-03-29 17:12:32 -07:00
• 1ac1dd9bab fix typo which removed call to candidate.setFullVia(), should fix failing Precedence3SelfTest Noah Levitt 2013-03-29 13:17:58 -07:00
• dcbbec4bbb fix for HER-2035 xml representation of job url can be invalid Noah Levitt 2013-03-29 12:16:07 -07:00
• 5fdf5713a2 Merge branch 'master' of github.com:internetarchive/heritrix3 Noah Levitt 2013-03-28 19:10:19 -07:00
• f988a26e10 move some code into EngineModel constructor and restore reverse chronological ordering of crawl jobs Noah Levitt 2013-03-28 19:09:14 -07:00
• 6a2b4a4a76 Merge branch 'master' into freemarker Noah Levitt 2013-03-28 18:01:16 -07:00
• 8ba69e6706 put .settings in .gitignore after all Noah Levitt 2013-03-28 17:53:20 -07:00
• 3665e129f3 put .settings in .gitignore after all Noah Levitt 2013-03-28 17:52:13 -07:00
• 2df2427e55 Dependency Fixing: Added LaxHttpParser which can be used explicitly when using the lax header parsing functionality. Added HttpParser which just extends LaxHttpParser in the original httpclient package to avoid breaking compatibility with code which may still reference HttpParser. Explicit commons code changed to use LaxHttpParser in arc/warc record reading Ilya Kreymer 2013-03-27 18:29:55 -07:00
• a08d83a370 Merge branch 'master' of github.com:internetarchive/heritrix3 Noah Levitt 2013-03-27 14:48:32 -07:00
• 69eb124b5d add test for handling of bad http headers (to make sure we don't break this when we get rid of archive-overlay) Noah Levitt 2013-03-27 14:48:21 -07:00
• a4a3ce60cd fix failing ContentDigestHistoryTest Kenji Nagahashi 2013-03-26 15:06:31 -07:00
• 5d8fe0443d Merge branch 'master' into HER-2031 Noah Levitt 2013-03-23 19:16:05 -07:00
• 0631123783 close recorders on failure to avoid exception from retry cycle within httpclient (restore functionality from fcece02646 to work with current httpcomponents Noah Levitt 2013-03-20 16:59:06 -07:00
• 75abd2509f bit of a hack to support shouldFetchBodyRule, given httpcomponents limitation discussed at https://issues.apache.org/jira/browse/HTTPCORE-325 Noah Levitt 2013-03-20 16:17:34 -07:00
• 956024ec77 update httpclient dependency Noah Levitt 2013-03-20 16:11:47 -07:00
• 1993c4388c Merge branch 'master' into hc43 Noah Levitt 2013-03-20 14:28:13 -07:00
• 042ed066b3 add more eclipse stuff to .gitignore Noah Levitt 2013-03-20 14:27:34 -07:00
• 8fd361e518 update source/target version to 1.6 since we use 1.6 features, specifically @Override for methods only implementing an interface (not sure why maven wasn't enforcing this); also remove maven-antrun-plugin created timestamp.txt, doesn't seem to be used anywhere Noah Levitt 2013-03-20 14:25:22 -07:00
• c509b6e552 remove more eclipse stuff, people like to manage their own Noah Levitt 2013-03-20 13:39:28 -07:00
• 654c106d19 remove eclipse stuff and add to .gitignore (except .settings which has code formatting config) Noah Levitt 2013-03-20 12:55:45 -07:00
• 8a14b40564 Merge branch 'master' into hc43 Noah Levitt 2013-03-20 12:39:58 -07:00
• 79422099b4 remove old profiling stuff Noah Levitt 2013-03-20 12:35:41 -07:00
• b4bff3ffb9 new method RecordingInputStream.readToEndOfContent(), because now we can't rely on stopping right after the headers, which is an assumption of readFullyOrUntil() Noah Levitt 2013-03-19 18:24:02 -07:00
• 9e054dd1f8 detect http message body begin in RecordingOutputStream, see https://issues.apache.org/jira/browse/HTTPCORE-325 Noah Levitt 2013-03-19 14:18:11 -07:00
• 0be1b8e318 partly done not working update to latest httpcomponents Noah Levitt 2013-03-19 13:01:35 -07:00
• 52ad4395d9 Merge branch 'master' into hc43 Noah Levitt 2013-03-19 13:00:27 -07:00
• 75bb19c4ea switch .classpath from archive-commons to ia-web-commons Noah Levitt 2013-03-19 11:25:13 -07:00
• 12da824316 Merge branch 'master' into hc43 Noah Levitt 2013-03-18 23:32:51 -07:00
• 9fa198944a depend on new "ia-web-commons" instead of "archive-commons" Noah Levitt 2013-03-18 23:32:05 -07:00
• b0c6619f80 Merge branch 'master' of github.com:internetarchive/heritrix3 Noah Levitt 2013-03-18 20:02:03 -07:00
• 0c9e2a1f4d ContentDigestHistory{Loader,Storer} - ignore uris with empty content body (deduping that is counterproductive) Noah Levitt 2013-03-18 20:01:52 -07:00
• ff36546eda fix serverless whois url example in comment Noah Levitt 2013-03-18 14:03:39 -07:00
• 2f4d437734 Beans.ftl Fixed a bug that showed objects with no properties as null instead of displaying the class name Job.ftl Fixed a bug that prevented the configuration referenced paths from displaying Script.ftl Fixed a bug that rendered extra whitespace before and after the script block Adam Miller 2013-03-15 16:06:06 -07:00
• c5c54ac138 comment-out debug output gojomo 2013-03-12 11:42:47 -07:00
• 7c8295ee82 checkpoint support gojomo 2013-03-12 11:42:29 -07:00
• a844878e7d Merge branch 'simple_logins' into HER-2031 gojomo 2013-03-12 01:16:31 -07:00
• 3f8590fe44 when found in CrawlURI, add WARC-Simple-Form-Province-Status header gojomo 2013-03-11 20:57:41 -07:00
• bf9cab8eb2 when POST/submit-data requested, alter request method accordingly gojomo 2013-03-11 20:57:09 -07:00
• bbac7d7320 per (overlay) settings, spawn high-priority POST when appropriate; annotate/add-headers to describe gojomo 2013-03-11 20:56:31 -07:00
• ed639c8342 extract HTMLForm INPUT details gojomo 2013-03-11 20:55:20 -07:00
• 94a61b4fb7 retain offsets of discovered FORMs to aid later processing gojomo 2013-03-11 20:53:41 -07:00
• 6b0abae8ee new constants, hop-type; refactoring to help support form-submissions gojomo 2013-03-11 20:52:59 -07:00
• eb0760a37e refactor for runCandidatesChain; also run outCandidates gojomo 2013-03-11 20:50:18 -07:00
• 0e3a2ec80a CrawlJobModel.java Adding check for application context Adam Miller 2013-03-04 16:34:18 -08:00
• b7b7953dc9 EngineResource.java Pulling individual values for necessary data from crawl jobs instead of building the full CrawlJobModel CrawlJobModel.java Moving some data from methods into fields so they are exposed in the XML representation EngineModel.java Removing full building of CrawlJobModel. Handling instead in EngineResource Job.ftl Updating fields that relied on CrawlJobModel methods which were moved to fields. Adam Miller 2012-09-26 16:36:09 -07:00
• 03f4f381af JobResource.java Removing temporary template loading. Finishing converting XML and HTML generation to use the same data model CrawlJobModel.java Converting to same Map based reports used by XML API EngineModel.java Converting to same Map based reports used by XML API Engine.ftl Creating template for Map based reports Job.ftl Creating template for Map based reports Adam Miller 2012-09-25 18:07:14 -07:00
• ef7fd37a0a BeanBrowseResource.java Removing obsolete method: makePresentableMap() Adam Miller 2012-09-12 14:26:31 -07:00
• d26a712211 Fixed bad merge conflict Adam Miller 2012-09-11 16:33:39 -07:00
• 60d00e3ef3 Converted data models to be usable for conversion to HTML and XML. Updated Models and freemarker templates to match XML naming Adam Miller 2012-09-10 17:03:49 -07:00
• 50e49b6399 Removed commented out code. Adam Miller 2012-09-10 17:02:27 -07:00
• 7426c78d73 Removed temporary template loading from file system. Removed unused method: getAvailableActions Adam Miller 2012-09-10 17:00:54 -07:00
• d545899fd2 Removed temporary template loading from file system Adam Miller 2012-09-10 16:59:56 -07:00
• 929f3cb489 Scripts and Beans converted to freemarker Adam Miller 2012-08-24 15:25:45 -07:00
• 783d48b49c Removing old commented out code Adam Miller 2012-07-26 16:42:56 -07:00
• a32f00c2b3 Finishing /engine and /engine/job/{job} Adam Miller 2012-07-26 16:36:23 -07:00
• e4ad60dda6 Beginning conversion to freemarker templates. /engine updated Adam Miller 2012-07-02 10:14:21 -07:00
• 01072b4b46 support mixing fetch histories from multiple sources (hbase, WBM). Kenji Nagahashi 2013-03-06 17:28:38 -08:00
• 8cbb84b8df Merge branch 'master' of github.com:internetarchive/heritrix3 Noah Levitt 2013-02-22 15:43:30 -08:00
• fb758e53e3 Avoid pulling in hadoop-core.jar and its dependencies; add plugin versions to get rid of warnings Noah Levitt 2013-02-22 15:43:15 -08:00
• 0841840baf Fix for HER-2032 - test stream for compression; add tests with and without compression Noah Levitt 2013-02-05 18:45:14 -08:00
• 60d93cbb5b Remove obsolete <scm> section that pointed to old svn repository Noah Levitt 2013-02-05 18:37:56 -08:00
• 878e3b1afe Remove spurious "throws" declaration Noah Levitt 2013-02-05 18:37:22 -08:00
• bda50a56b1 Merge pull request #19 from travisfw/bean-bindings Noah Levitt 2013-01-24 13:59:29 -08:00
• bf82ea3524 Rename to README.md so github treats it as markdown Noah Levitt 2013-01-24 13:57:02 -08:00
• c26e1c9732 Rename to README.md so github treats it as markdown Noah Levitt 2013-01-24 13:56:20 -08:00