summarization

Currently Crawler does not let JavaScript run on crawled pages. This leads to issues for websites that rely on fully dynamically loaded content that is not present in the DOM at loading time, but is fetched with JS immediately after.

We should investigate how to allow JavaScript to run on the crawled pages with Selenium and add a switch to the Crawler that allows users to do this.

Based on

Intro

I am getting TypeError: can not serialize 'BaseTextRank' object when trying to use spaCy's multiprocessing in nlp.pipe with a textrank pipeline component.

Sorry if this a known/expected feature/limitation - I couldn't find anything by searching repo. I generally find (spaCy's) multiprocessing a bit temperamental anyhow, but this seems to just not work.

_PS. thanks for all

Need help for retraining and cross validation and see if the ROUGE score matches exactly (or better) with the numbers reported in the paper.
I just train for 500k iteration (with batch size 8) with pointer generation enabled + coverage loss disabled and next 100k iteration (with batch size 8) with pointer generation enabled + coverage loss enabled.

It would be great if someone can help re-r

A next step for better generation is to implement a beam search for the generation. An example of it can be seen on the huggingface repo here, and this would need adding such a function to the GenerativeT5 model in onnxt5/models.py