Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nightly integration tests #1664

Merged
merged 29 commits into from Dec 16, 2022
Merged

Nightly integration tests #1664

merged 29 commits into from Dec 16, 2022

Conversation

anton-l
Copy link
Member

@anton-l anton-l commented Dec 12, 2022

In an effort to speed up our slow/push tests, this PR:

  • adds a @nightly decorator for tests that can be enabled with RUN_NIGHTLY=1
  • suggests num_inference_steps=3 for all @slow tests
  • moves end-to-end image reference-based @slow tests to @nightly
  • suggests num_inference_steps~50 for all @nightly tests

Nightly tests will be run daily at midnight by our CI.

@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Dec 12, 2022

The documentation is not available anymore as the PR was closed or merged.

@anton-l anton-l marked this pull request as ready for review Dec 16, 2022
@anton-l anton-l changed the title [WIP] Nightly integration tests Nightly integration tests Dec 16, 2022
@anton-l anton-l changed the title Nightly integration tests [WIP] Nightly integration tests Dec 16, 2022
@@ -212,211 +213,215 @@ def test_stable_diffusion_img2img_num_images_per_prompt(self):

@slow
@require_torch_gpu
class StableDiffusionImg2ImgPipelineIntegrationTests(unittest.TestCase):
class StableDiffusionImg2ImgPipelineSlowTests(unittest.TestCase):
Copy link
Member Author

@anton-l anton-l Dec 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Img2Img is the most flaky, max_diff > 1e-4 even with 3 steps. If I won't find the cause today, will raise the tolerances

Copy link
Member

@patrickvonplaten patrickvonplaten Dec 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good 1e-3 is totally fine. FYI in transformers we only have 5e-3 for some one-forward pass tests

Copy link
Member

@patrickvonplaten patrickvonplaten left a comment

Super cool! Looks all good to me :-)

Also definitely enough to start with stable diffusion for now and then maybe open some issue for things that should changed after :-)

anton-l and others added 2 commits Dec 16, 2022
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Copy link
Contributor

@pcuenca pcuenca left a comment

Looking great! Just fixed a typo in the comment for RUN_NIGHTLY

src/diffusers/utils/testing_utils.py Outdated Show resolved Hide resolved
@anton-l anton-l changed the title [WIP] Nightly integration tests Nightly integration tests Dec 16, 2022
@anton-l anton-l merged commit 086c7f9 into main Dec 16, 2022
7 checks passed
@anton-l anton-l deleted the nightly-tests branch Dec 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants