Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add IntervalDescriptor and change DurationConverter to use Interval datatype #653

Merged
merged 8 commits into from Aug 3, 2020

Conversation

@sarahcaseybot
Copy link
Collaborator

@sarahcaseybot sarahcaseybot commented Jun 25, 2020

This change is Reviewable

Copy link
Member

@CydeWeys CydeWeys left a comment

Reviewable status: 0 of 15 files reviewed, 2 unresolved discussions (waiting on @CydeWeys and @sarahcaseybot)


core/src/main/java/google/registry/persistence/converter/DurationConverter.java, line 27 at r1 (raw file):

/**
 * JPA converter to for storing/retrieving {@link org.joda.time.Duration} objects. Note: this only
 * converts the values for days, hours, minutes, and seconds

The Note: can be in a separate <p>.


core/src/main/java/google/registry/persistence/converter/DurationConverter.java, line 30 at r1 (raw file):

 */
@Converter(autoApply = true)
public class DurationConverter implements AttributeConverter<Duration, Object> {

Why Object instead of PGInterval?

@CydeWeys CydeWeys requested review from hstonec and mindhog Jun 25, 2020
Copy link
Member

@CydeWeys CydeWeys left a comment

+reviewer:@hstonec +reviewer:@mindhog

Adding some other reviewers who might know more about Cloud SQL converters than I do.

Reviewable status: 0 of 15 files reviewed, 2 unresolved discussions (waiting on @CydeWeys, @hstonec, @mindhog, and @sarahcaseybot)

Copy link
Collaborator

@hstonec hstonec left a comment

Reviewable status: 0 of 15 files reviewed, 6 unresolved discussions (waiting on @hstonec, @mindhog, and @sarahcaseybot)


core/src/main/java/google/registry/persistence/converter/IntervalDescriptor.java, line 41 at r1 (raw file):

public static final int COLUMN_TYPE = 1426407511;

I'm just curious how this number was selected?


db/src/main/resources/sql/flyway/V20__add_relock_duration.sql, line 15 at r1 (raw file):

ALTER TABLE "RegistryLock" ADD COLUMN relock_duration interval;

we should create a new flyway file instead of modifying the existing ones.


db/src/main/resources/sql/schema/db-schema.sql.generated, line 386 at r1 (raw file):

relock_duration interval,

do we have data in this table? if so, we may want to follow our procedure to change the column.


db/src/main/resources/sql/schema/nomulus.golden.sql, line 593 at r1 (raw file):

relock_duration interval

do we have data in this table? if so, we may want to follow our procedure to change the column.

Copy link
Member

@mindhog mindhog left a comment

Reviewed 15 of 15 files at r1.
Reviewable status: all files reviewed, 10 unresolved discussions (waiting on @sarahcaseybot)


core/src/main/java/google/registry/persistence/converter/DurationConverter.java, line 36 at r1 (raw file):

  public Object convertToDatabaseColumn(@Nullable Duration duration) {
    if (duration == null) {
      return new PGInterval();

Don't we typically return null in this case? Does that not work for interval columns?


core/src/main/java/google/registry/persistence/converter/DurationConverter.java, line 43 at r1 (raw file):

    interval.setHours(period.getHours());
    interval.setMinutes(period.getMinutes());
    interval.setSeconds(period.getSeconds());

I suspect you're going to lose millisecond precision, here. It looks like joda Duration has seconds and milliseconds as integers whereas PGInterval has double-precision seconds. if so, you'll want to convert seconds + milliseconds / 1000 to a double.


core/src/main/java/google/registry/schema/domain/RegistryLock.java, line 217 at r1 (raw file):

  }

  /** Returns true iff the lock was requested >= 1 hour ago and has not been verified. */

This seems orthogonal to this PR?


core/src/test/java/google/registry/persistence/converter/DurationConverterTest.java, line 55 at r1 (raw file):

    TestEntity persisted =
        jpaTm().transact(() -> jpaTm().getEntityManager().find(TestEntity.class, "id"));
    assertThat(persisted.duration.getMillis()).isEqualTo(Duration.standardDays(6).getMillis());

Per my earlier statement, you should do this with a duration of 6 days + some odd number of millis

Copy link
Collaborator

@hstonec hstonec left a comment

Reviewable status: all files reviewed, 10 unresolved discussions (waiting on @sarahcaseybot)


db/src/main/resources/sql/schema/nomulus.golden.sql, line 593 at r1 (raw file):

Previously, hstonec (Shicong Huang) wrote…
relock_duration interval

do we have data in this table? if so, we may want to follow our procedure to change the column.

@weiminyu

When we need to change the type of the relock_duration column, do we have to follow a procedure like adding a new column with the new type, change the code to use the new column, then delete the old column?

Given that we haven't launched registry lock yet, is there a simpler way to change the schema?

@sarahcaseybot sarahcaseybot force-pushed the sarahcaseybot:addIntervalDataType branch from 957e26c to 69f1c2c Jun 30, 2020
Copy link
Member

@mindhog mindhog left a comment

LGTM modulo concerns expressed by Ben & Shicong.

Reviewed 5 of 5 files at r2.
Reviewable status: all files reviewed, 10 unresolved discussions (waiting on @sarahcaseybot)

@sarahcaseybot sarahcaseybot force-pushed the sarahcaseybot:addIntervalDataType branch from c52b9f8 to 4cbc86a Jul 1, 2020
Copy link
Collaborator Author

@sarahcaseybot sarahcaseybot left a comment

Reviewable status: 13 of 14 files reviewed, 10 unresolved discussions (waiting on @CydeWeys, @hstonec, @mindhog, and @weiminyu)


core/src/main/java/google/registry/persistence/converter/DurationConverter.java, line 27 at r1 (raw file):

Previously, CydeWeys (Ben McIlwain) wrote…

The Note: can be in a separate <p>.

Done.


core/src/main/java/google/registry/persistence/converter/DurationConverter.java, line 30 at r1 (raw file):

Previously, CydeWeys (Ben McIlwain) wrote…

Why Object instead of PGInterval?

Done.


core/src/main/java/google/registry/persistence/converter/DurationConverter.java, line 36 at r1 (raw file):

Previously, mindhog (Michael Muller) wrote…

Don't we typically return null in this case? Does that not work for interval columns?

Returning null causes a PSQLException so I changed converter to use an empty interval when creating a database column, and turn an empty interval into a null duration when converting it back


core/src/main/java/google/registry/persistence/converter/DurationConverter.java, line 43 at r1 (raw file):

Previously, mindhog (Michael Muller) wrote…

I suspect you're going to lose millisecond precision, here. It looks like joda Duration has seconds and milliseconds as integers whereas PGInterval has double-precision seconds. if so, you'll want to convert seconds + milliseconds / 1000 to a double.

Done.


core/src/main/java/google/registry/persistence/converter/IntervalDescriptor.java, line 41 at r1 (raw file):

Previously, hstonec (Shicong Huang) wrote…
public static final int COLUMN_TYPE = 1426407511;

I'm just curious how this number was selected?

I believe that is the number for the Interval type, but I changed it to use JAVA_OBJECT which is 2000


core/src/main/java/google/registry/schema/domain/RegistryLock.java, line 217 at r1 (raw file):

Previously, mindhog (Michael Muller) wrote…

This seems orthogonal to this PR?

Done.


core/src/test/java/google/registry/persistence/converter/DurationConverterTest.java, line 55 at r1 (raw file):

Previously, mindhog (Michael Muller) wrote…

Per my earlier statement, you should do this with a duration of 6 days + some odd number of millis

Done.


db/src/main/resources/sql/flyway/V20__add_relock_duration.sql, line 15 at r1 (raw file):

Previously, hstonec (Shicong Huang) wrote…
ALTER TABLE "RegistryLock" ADD COLUMN relock_duration interval;

we should create a new flyway file instead of modifying the existing ones.

Done.


db/src/main/resources/sql/schema/db-schema.sql.generated, line 386 at r1 (raw file):

Previously, hstonec (Shicong Huang) wrote…
relock_duration interval,

do we have data in this table? if so, we may want to follow our procedure to change the column.

dropped the column and added a new one


db/src/main/resources/sql/schema/nomulus.golden.sql, line 593 at r1 (raw file):

Previously, hstonec (Shicong Huang) wrote…

@weiminyu

When we need to change the type of the relock_duration column, do we have to follow a procedure like adding a new column with the new type, change the code to use the new column, then delete the old column?

Given that we haven't launched registry lock yet, is there a simpler way to change the schema?

Talked to Weimin and Gus, since it hasn't been pushed to production yet, I dropped the column and added a new one.

Copy link
Member

@CydeWeys CydeWeys left a comment

Reviewable status: 13 of 14 files reviewed, 10 unresolved discussions (waiting on @hstonec, @mindhog, @sarahcaseybot, and @weiminyu)


core/src/main/java/google/registry/persistence/converter/DurationConverter.java, line 28 at r3 (raw file):

 * JPA converter to for storing/retrieving {@link org.joda.time.Duration} objects.
 *
 * <p>Note: this only converts the values for days, hours, minutes, and seconds

Is this comment still accurate? Why are we throwing away years and milliseconds? We use both of those in our codebase.


core/src/main/java/google/registry/persistence/converter/DurationConverter.java, line 41 at r3 (raw file):

    PGInterval interval = new PGInterval();
    Period period = new Period(duration);
    interval.setDays(period.getDays());

Why no setYears? It seems like we're just setting ourselves up for bugs down the line by not making the converter handle it, when handling it seems so simple.

@sarahcaseybot sarahcaseybot force-pushed the sarahcaseybot:addIntervalDataType branch from 4cbc86a to 18c25d8 Jul 9, 2020
Copy link
Collaborator Author

@sarahcaseybot sarahcaseybot left a comment

Reviewable status: 10 of 14 files reviewed, 10 unresolved discussions (waiting on @CydeWeys, @hstonec, @mindhog, and @weiminyu)


core/src/main/java/google/registry/persistence/converter/DurationConverter.java, line 28 at r3 (raw file):

Previously, CydeWeys (Ben McIlwain) wrote…

Is this comment still accurate? Why are we throwing away years and milliseconds? We use both of those in our codebase.

Removed


core/src/main/java/google/registry/persistence/converter/DurationConverter.java, line 41 at r3 (raw file):

Previously, CydeWeys (Ben McIlwain) wrote…

Why no setYears? It seems like we're just setting ourselves up for bugs down the line by not making the converter handle it, when handling it seems so simple.

added years and months

Copy link
Member

@CydeWeys CydeWeys left a comment

Reviewable status: 10 of 14 files reviewed, 11 unresolved discussions (waiting on @hstonec, @mindhog, @sarahcaseybot, and @weiminyu)


core/src/main/java/google/registry/persistence/converter/DurationConverter.java, line 41 at r3 (raw file):

Previously, sarahcaseybot wrote…

added years and months

I'm not seeing a test that all of this works? I see a test that does 6 days plus 7 millis, but what about years, months, hours, minutes, and seconds?


core/src/test/java/google/registry/persistence/converter/DurationConverterTest.java, line 50 at r4 (raw file):

  @Test
  public void testRoundTrip() {

Can you add a test that


core/src/test/java/google/registry/persistence/converter/DurationConverterTest.java, line 61 at r4 (raw file):

  @Entity(name = "TestEntity") // Override entity name to avoid the nested class reference.
  @EntityForTesting
  public static class TestEntity extends ImmutableObject {

You should probably rename this to something like DurationTestEntity to prevent potential entity name conflicts causing flaky tests (which has happened a lot in the past).

@sarahcaseybot sarahcaseybot force-pushed the sarahcaseybot:addIntervalDataType branch from 18c25d8 to a284a0c Jul 21, 2020
Copy link
Collaborator Author

@sarahcaseybot sarahcaseybot left a comment

Reviewable status: 6 of 14 files reviewed, 11 unresolved discussions (waiting on @CydeWeys, @hstonec, @mindhog, and @weiminyu)


core/src/main/java/google/registry/persistence/converter/DurationConverter.java, line 41 at r3 (raw file):

Previously, CydeWeys (Ben McIlwain) wrote…

I'm not seeing a test that all of this works? I see a test that does 6 days plus 7 millis, but what about years, months, hours, minutes, and seconds?

Added a test. Realized that years and months do not work since a duration is simply an amount of millis, and since years and months are not a standard size, they cannot be accurately converted to years and months values. Instead, these values will be represented int he days field.


core/src/test/java/google/registry/persistence/converter/DurationConverterTest.java, line 50 at r4 (raw file):

Previously, CydeWeys (Ben McIlwain) wrote…

Can you add a test that

Done.


core/src/test/java/google/registry/persistence/converter/DurationConverterTest.java, line 61 at r4 (raw file):

Previously, CydeWeys (Ben McIlwain) wrote…

You should probably rename this to something like DurationTestEntity to prevent potential entity name conflicts causing flaky tests (which has happened a lot in the past).

Done.

@sarahcaseybot sarahcaseybot force-pushed the sarahcaseybot:addIntervalDataType branch from a284a0c to da7e0a5 Jul 23, 2020
@sarahcaseybot sarahcaseybot requested a review from CydeWeys Jul 23, 2020
Copy link
Member

@mindhog mindhog left a comment

Reviewed 2 of 3 files at r3, 8 of 8 files at r5.
Reviewable status: all files reviewed, 8 unresolved discussions (waiting on @CydeWeys, @hstonec, @mindhog, and @weiminyu)

Copy link
Member

@mindhog mindhog left a comment

LGTM

Reviewable status: all files reviewed, 8 unresolved discussions (waiting on @CydeWeys, @hstonec, @mindhog, and @weiminyu)

Copy link
Member

@CydeWeys CydeWeys left a comment

Reviewable status: all files reviewed, 9 unresolved discussions (waiting on @CydeWeys, @hstonec, @mindhog, @sarahcaseybot, and @weiminyu)


core/src/main/java/google/registry/persistence/converter/DurationConverter.java, line 41 at r3 (raw file):

Previously, sarahcaseybot wrote…

Added a test. Realized that years and months do not work since a duration is simply an amount of millis, and since years and months are not a standard size, they cannot be accurately converted to years and months values. Instead, these values will be represented int he days field.

Are you still working on adding this? I don't see it in here.


core/src/main/java/google/registry/persistence/converter/DurationConverter.java, line 25 at r5 (raw file):

import org.postgresql.util.PGInterval;

/** JPA converter to for storing/retrieving {@link org.joda.time.Duration} objects. * */

The Javadoc needs more explanation as to how exactly all the different parts of a Joda Time Duration is stored, seeing as how there appears to be a mismatch between that and how PGInterval works.

@sarahcaseybot sarahcaseybot force-pushed the sarahcaseybot:addIntervalDataType branch from da7e0a5 to 1c0840f Jul 27, 2020
Copy link
Collaborator Author

@sarahcaseybot sarahcaseybot left a comment

Reviewable status: 9 of 14 files reviewed, 9 unresolved discussions (waiting on @CydeWeys, @hstonec, @mindhog, and @weiminyu)


core/src/main/java/google/registry/persistence/converter/DurationConverter.java, line 41 at r3 (raw file):

Previously, CydeWeys (Ben McIlwain) wrote…

Are you still working on adding this? I don't see it in here.

I added values for days, hours, minutes, and seconds in the test roundtrip test


core/src/main/java/google/registry/persistence/converter/DurationConverter.java, line 25 at r5 (raw file):

Previously, CydeWeys (Ben McIlwain) wrote…

The Javadoc needs more explanation as to how exactly all the different parts of a Joda Time Duration is stored, seeing as how there appears to be a mismatch between that and how PGInterval works.

Done.


core/src/main/java/google/registry/persistence/converter/IntervalDescriptor.java, line 41 at r1 (raw file):

Previously, sarahcaseybot wrote…

I believe that is the number for the Interval type, but I changed it to use JAVA_OBJECT which is 2000

Done.

Copy link
Member

@CydeWeys CydeWeys left a comment

Reviewable status: 9 of 14 files reviewed, 7 unresolved discussions (waiting on @hstonec, @mindhog, @sarahcaseybot, and @weiminyu)


core/src/main/java/google/registry/persistence/converter/DurationConverter.java, line 31 at r6 (raw file):

 * can be converted into a PGInterval, but only for the fields that have a standard number of
 * milliseconds. Therefore, there is no way to populate the months or years field of a PGInterval
 * and be confident that it is representing the exact number of milliseconds it was intended to

There's one final part missing here in the explanation, which is what happens when the Duration does include months or years. Are they ignored? Just stored as days using some flat conversion factor? Is an exception thrown? (Maybe an exception should be thrown rather than silently ignoring them?)


core/src/test/java/google/registry/persistence/converter/DurationConverterTest.java, line 32 at r6 (raw file):

/** Unit tests for {@link DurationConverter}. */
public class DurationConverterTest {

Still needs tests for what happens when the Duration includes months and/or years.

Copy link
Member

@mindhog mindhog left a comment

Reviewed 4 of 5 files at r6.
Reviewable status: 13 of 14 files reviewed, 8 unresolved discussions (waiting on @hstonec, @mindhog, @sarahcaseybot, and @weiminyu)


db/src/main/resources/sql/flyway/V42__update_relock_duration_type.sql, line 1 at r6 (raw file):

-- Copyright 2020 The Nomulus Authors. All Rights Reserved.

I checked in V42 yesterday, you might want to hold off on rebasing and further renaming this file until you're ready to commit.

Copy link
Member

@mindhog mindhog left a comment

Reviewed 1 of 5 files at r6.
Reviewable status: all files reviewed, 8 unresolved discussions (waiting on @hstonec, @mindhog, @sarahcaseybot, and @weiminyu)

Copy link
Member

@CydeWeys CydeWeys left a comment

Reviewable status: all files reviewed, 8 unresolved discussions (waiting on @hstonec, @mindhog, @sarahcaseybot, and @weiminyu)


core/src/main/java/google/registry/persistence/converter/DurationConverter.java, line 31 at r6 (raw file):

Previously, CydeWeys (Ben McIlwain) wrote…

There's one final part missing here in the explanation, which is what happens when the Duration does include months or years. Are they ignored? Just stored as days using some flat conversion factor? Is an exception thrown? (Maybe an exception should be thrown rather than silently ignoring them?)

Per out of band chat, java.util.Duration doesn't have any larger part than a day, so ignore that, but we should add a test for a large number of days (more than a year) just to rule out the possibility that anything weird is happening there, and also, Duration does support down to nanoseconds, so we should get that working and add a test of that.

Copy link
Member

@CydeWeys CydeWeys left a comment

Reviewable status: all files reviewed, 8 unresolved discussions (waiting on @hstonec, @mindhog, @sarahcaseybot, and @weiminyu)


core/src/main/java/google/registry/persistence/converter/DurationConverter.java, line 31 at r6 (raw file):

Previously, CydeWeys (Ben McIlwain) wrote…

Per out of band chat, java.util.Duration doesn't have any larger part than a day, so ignore that, but we should add a test for a large number of days (more than a year) just to rule out the possibility that anything weird is happening there, and also, Duration does support down to nanoseconds, so we should get that working and add a test of that.

Regarding nanoseconds, never mind. I was looking at the wrong Duration. We use JodaTime, which only supports granularity down to millis. So we don't need to handle anything finer than that. Also, TIL that Java's built-in Duration has 6 orders of magnitude better resolution than JodaTime.

@sarahcaseybot sarahcaseybot force-pushed the sarahcaseybot:addIntervalDataType branch from 1c0840f to da62a47 Jul 30, 2020
Copy link
Collaborator Author

@sarahcaseybot sarahcaseybot left a comment

Reviewable status: 7 of 14 files reviewed, 8 unresolved discussions (waiting on @CydeWeys, @hstonec, @mindhog, and @weiminyu)


core/src/main/java/google/registry/persistence/converter/DurationConverter.java, line 31 at r6 (raw file):

Previously, CydeWeys (Ben McIlwain) wrote…

Regarding nanoseconds, never mind. I was looking at the wrong Duration. We use JodaTime, which only supports granularity down to millis. So we don't need to handle anything finer than that. Also, TIL that Java's built-in Duration has 6 orders of magnitude better resolution than JodaTime.

Done.


core/src/test/java/google/registry/persistence/converter/DurationConverterTest.java, line 32 at r6 (raw file):

Previously, CydeWeys (Ben McIlwain) wrote…

Still needs tests for what happens when the Duration includes months and/or years.

This is not necessary since there is no way for Duration to have months or years


db/src/main/resources/sql/schema/db-schema.sql.generated, line 386 at r1 (raw file):

Previously, sarahcaseybot wrote…

dropped the column and added a new one

Done.


db/src/main/resources/sql/schema/nomulus.golden.sql, line 593 at r1 (raw file):

Previously, sarahcaseybot wrote…

Talked to Weimin and Gus, since it hasn't been pushed to production yet, I dropped the column and added a new one.

Done.


db/src/main/resources/sql/flyway/V42__update_relock_duration_type.sql, line 1 at r6 (raw file):

Previously, mindhog (Michael Muller) wrote…

I checked in V42 yesterday, you might want to hold off on rebasing and further renaming this file until you're ready to commit.

Done.

Copy link
Member

@CydeWeys CydeWeys left a comment

Reviewable status: 7 of 14 files reviewed, 7 unresolved discussions (waiting on @hstonec, @mindhog, @sarahcaseybot, and @weiminyu)


core/src/test/java/google/registry/persistence/converter/DurationConverterTest.java, line 69 at r7 (raw file):

    DurationTestEntity persisted =
        jpaTm().transact(() -> jpaTm().getEntityManager().find(DurationTestEntity.class, "id"));
    assertThat(persisted.duration.getMillis()).isEqualTo(testDuration.getMillis());

Hrmmm ... I just realized .... you could put in a Duration containing hours, minutes, seconds, etc., but when you get it back out, everything has been converted to millis.

So we need tests that explicitly verify that the days, hours, minutes, seconds, and millis that you put in are exactly what you get back out.

@sarahcaseybot
Copy link
Collaborator Author

@sarahcaseybot sarahcaseybot commented Aug 1, 2020


core/src/test/java/google/registry/persistence/converter/DurationConverterTest.java, line 69 at r7 (raw file):

Previously, CydeWeys (Ben McIlwain) wrote…

Hrmmm ... I just realized .... you could put in a Duration containing hours, minutes, seconds, etc., but when you get it back out, everything has been converted to millis.

So we need tests that explicitly verify that the days, hours, minutes, seconds, and millis that you put in are exactly what you get back out.

I don't agree with this. The joda Duration class is simply an amount of milliseconds. It does not store fields such as days, hours, minutes. It has methods for adding values such as days or hours, but all those methods do is multiply the number of milliseconds in that time type by the number of that time type. When getting days, hours, etc. from a duration, it is simply received from a division of the millis in the total duration by the time type that is being asked for. The only way I would think to write a test like that would be to convert to a period. But the way that a duration is converted to a period will not preserve the specific number of each timetype that had been initially added to the duration since those are not stored anywhere.

Copy link
Member

@CydeWeys CydeWeys left a comment

:lgtm:

Reviewable status: 7 of 14 files reviewed, 6 unresolved discussions (waiting on @hstonec, @mindhog, and @weiminyu)

@hstonec
hstonec approved these changes Aug 3, 2020
Copy link
Collaborator

@hstonec hstonec left a comment

:lgtm:

Reviewable status: 7 of 14 files reviewed, 2 unresolved discussions (waiting on @mindhog)

@sarahcaseybot sarahcaseybot merged commit 4ad7f97 into google:master Aug 3, 2020
6 of 7 checks passed
6 of 7 checks passed
code-review/reviewable 7 files, 2 discussions left (mindhog)
Details
LGTM analysis: JavaScript No code changes detected
Details
LGTM analysis: Python No code changes detected
Details
LGTM analysis: Java No new or fixed alerts
Details
cla/google All necessary CLAs are signed
kokoro-foss Kokoro build finished
Details
kokoro-internal Kokoro build finished
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants
You can’t perform that action at this time.