Add administrative interface to invoker and controller to reconfigure runtimes #4790

ningyougang · 2020-01-07T03:18:33Z

Sometimes, admin may want to reinitalize the runtime config, e.g. nodejs:10 prewarm container number is less, lead to cold start, in order to handle user's request as soon as possible, admin may want to reinitalize the runtime configuration to increase the nodejs:10 prewarm containers.
And admin may want to reinitalize the runtime config on some limited invokers, this patch includes below changes

config runtime via controller to all managed invokers(or some limited inovkers which included in managed inovkers)
config runtime via invoker directly
get runtime info via invoker directly
add documents
test cases

Description

Related issue and scope

I opened an issue to propose and discuss this change (#????)

My changes affect the following components

Types of changes

Bug fix (generally a non-breaking change which closes an issue).
Enhancement or new feature (adds new functionality).
Breaking change (a bug fix or enhancement which changes existing behavior).

Checklist:

I signed an Apache CLA.
I reviewed the style guides and followed the recommendations (Travis CI will check :).
I added tests to cover my changes.
My changes require further changes to the documentation.
I updated the documentation where necessary.

ningyougang · 2020-01-07T03:38:51Z

core/controller/src/main/scala/org/apache/openwhisk/core/controller/Controller.scala

+      extractCredentials {
+        case Some(BasicHttpCredentials(username, password)) =>
+          if (username == controllerUsername && password == controllerPassword) {
+            entity(as[String]) { runtime =>


Usually, entity(as[Runtimes]) may be better, but if we apply this, need to change Runtimes to support convert json to entity Runtimes, and on the other hand, runtime.json's format doesn't match Runtimes entity, need to change a lot if use entity(as[Runtimes]) to receive the data.

Fortunately，we can reuse openwhisk itself's initialize method, just convert the runtime string to Runtimes.

So here, i think pass runtime string would be ok.

ningyougang · 2020-01-07T03:40:39Z

core/controller/src/main/scala/org/apache/openwhisk/core/controller/Controller.scala

+                logging.error(this, s"Received invalid runtimes manifest")
+                complete(s"Received invalid runtimes manifest")
+              } else {
+                parameter('limit.?) { limit =>


Support passed limit invokers, e.g. ?limit=0:1 ( sent to invoker0, invoker1 only)
And the config runtime request can be sent to some limited invokers which included in managed invokers as well.

style95 · 2020-01-07T12:20:09Z

common/scala/src/main/resources/reference.conf

@@ -27,7 +27,7 @@ whisk.spi {
  EntitlementSpiProvider = org.apache.openwhisk.core.entitlement.LocalEntitlementProvider
  AuthenticationDirectiveProvider = org.apache.openwhisk.core.controller.BasicAuthenticationDirective
  InvokerProvider = org.apache.openwhisk.core.invoker.InvokerReactive
-  InvokerServerProvider = org.apache.openwhisk.core.invoker.DefaultInvokerServer
+  InvokerServerProvider = org.apache.openwhisk.core.invoker.InvokerServer


This should be reverted.

DefaultInvokerServer is already reverted

style95 · 2020-01-07T12:21:50Z

core/controller/src/main/scala/org/apache/openwhisk/core/controller/Controller.scala

+                        val invokerArray = targetValue.split(":")
+                        val beginIndex = invokerArray(0).toInt
+                        val finishIndex = invokerArray(1).toInt
+                        if (finishIndex < beginIndex) {


Would be great to support 0:0 case as well.
Ansible is using the same way.

Yes, already support 0:0 case.

codecov-io · 2020-01-13T09:03:33Z

Codecov Report

Merging #4790 into master will decrease coverage by 6.86%.
The diff coverage is 41.78%.

@@            Coverage Diff             @@
##           master    #4790      +/-   ##
==========================================
- Coverage   82.35%   75.49%   -6.87%     
==========================================
  Files         198      199       +1     
  Lines        9021     9123     +102     
  Branches      353      379      +26     
==========================================
- Hits         7429     6887     -542     
- Misses       1592     2236     +644

Impacted Files	Coverage Δ
...la/org/apache/openwhisk/core/invoker/Invoker.scala	`71.66% <ø> (-0.47%)`	⬇️
.../apache/openwhisk/core/controller/Controller.scala	`0% <0%> (ø)`	⬆️
...che/openwhisk/core/loadBalancer/LoadBalancer.scala	`15.38% <0%> (-1.29%)`	⬇️
...e/loadBalancer/ShardingContainerPoolBalancer.scala	`73.91% <0%> (-2.93%)`	⬇️
...rg/apache/openwhisk/core/entity/ExecManifest.scala	`97.36% <100%> (+0.14%)`	⬆️
.../scala/org/apache/openwhisk/core/WhiskConfig.scala	`94.85% <100%> (+0.03%)`	⬆️
.../org/apache/openwhisk/core/connector/Message.scala	`77.61% <37.5%> (-2.55%)`	⬇️
...e/openwhisk/core/containerpool/ContainerPool.scala	`89.04% <44.44%> (-6.61%)`	⬇️
...pache/openwhisk/core/invoker/InvokerReactive.scala	`78.62% <64.4%> (-1.03%)`	⬇️
.../openwhisk/core/invoker/DefaultInvokerServer.scala	`75% <75%> (ø)`
... and 21 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 394e9f6...12201bb. Read the comment docs.

ningyougang · 2020-01-13T09:20:08Z

ansible/roles/controller/tasks/deploy.yml

@@ -203,6 +203,9 @@
      "CONFIG_whisk_db_activationsFilterDdoc": "{{ db_whisk_activations_filter_ddoc | default() }}"
      "CONFIG_whisk_userEvents_enabled": "{{ user_events | default(false) | lower }}"

+      "CONFIG_whisk_credentials_controller_username": "{{ controller.username }}"
+      "CONFIG_whisk_credentials_controller_password": "{{ controller.password }}"
+


If generate controller/invoker credentials to container's /conf/, the Standalone Tests run failed due to lack /conf/ under the build machine

On the other hand, couchdb credentials is passed via environment variable, so controller/invoker credentials can be passed via environment variable as well.

ningyougang · 2020-01-14T08:35:11Z

Already added test cases

rabbah

The limiting of a runtime to specific invokers is related to how invokers are partitioned for black box vs non-black box invokers (based on a % of active invokers in the system). I suggest splitting out the "limit" feature into a separate PR and having a mechanism for specifying distributions of runtimes to invokers and letting the load balancer calculate the partition.

That isn't powerful enough to when it is desirable to route to a specific invoker - is that a desired feature (vs limiting to a range of invokers active in the system)?

rabbah · 2020-01-26T16:01:24Z

core/controller/src/main/scala/org/apache/openwhisk/core/controller/Controller.scala

+              val execManifest = ExecManifest.initialize(runtime)
+              if (execManifest.isFailure) {
+                logging.error(this, s"Received invalid runtimes manifest")
+                complete(s"Received invalid runtimes manifest")


this completes the https request with status code 200 - is that what was intended?

Hi, almost forgot this patch :(
I changed like below

logging.info(this, s"received invalid runtimes manifest") complete(StatusCodes.BadRequest)

rabbah · 2020-01-26T16:01:24Z

core/controller/src/main/scala/org/apache/openwhisk/core/controller/Controller.scala

+            entity(as[String]) { runtime =>
+              val execManifest = ExecManifest.initialize(runtime)
+              if (execManifest.isFailure) {
+                logging.error(this, s"Received invalid runtimes manifest")


.error is for internal errors, where here this is user input error, i think .info is better.

Already changed to .info

rabbah · 2020-01-26T16:01:24Z

core/controller/src/main/scala/org/apache/openwhisk/core/controller/Controller.scala

+                parameter('limit.?) { limit =>
+                  limit match {
+                    case Some(targetValue) =>
+                      val pattern = "\\d+:\\d"


use

Suggested change

val pattern = "\\d+:\\d"

val pattern = """\d+:\d"""

using triple quotes allows you to use a regex without escape characters.

rabbah · 2020-01-26T16:01:24Z

core/controller/src/main/scala/org/apache/openwhisk/core/controller/Controller.scala

+          if (username == controllerCredentials.username && password == controllerCredentials.password) {
+            entity(as[String]) { runtime =>
+              val execManifest = ExecManifest.initialize(runtime)
+              if (execManifest.isFailure) {


scala nit: you can use a case match here instead.

I just follow other codes using execManifest.isFailure to judge whether success or fail

rabbah · 2020-01-26T16:01:24Z

core/controller/src/main/scala/org/apache/openwhisk/core/controller/Controller.scala

+                          complete(s"finishIndex can't be less than beginIndex")
+                        } else {
+                          val targetInvokers = (beginIndex to finishIndex).toList
+                          loadBalancer.sendRuntimeToInvokers(runtime, Some(targetInvokers))


is there a later validation to check that the invoker indexing is in range?

Yes, validate it in ShardingContainerPoolLoadbalacer.scala

/** send runtime to invokers*/ override def sendRuntimeToInvokers(runtime: String, targetInvokers: Option[List[Int]]): Unit = { val runtimeMessage = RuntimeMessage(runtime) schedulingState.managedInvokers.filter { manageInvoker => targetInvokers.getOrElse(schedulingState.managedInvokers.map(_.id.instance)).contains(manageInvoker.id.instance) } foreach { invokerHealth => val topic = s"invoker${invokerHealth.id.toInt}" messageProducer.send(topic, runtimeMessage).andThen { case Success(_) => logging.info(this, s"Successfully posted runtime to topic $topic") case Failure(_) => logging.error(this, s"Failed posted runtime to topic $topic") } } }

rabbah · 2020-01-26T16:01:24Z

core/controller/src/main/scala/org/apache/openwhisk/core/controller/Controller.scala

+                  limit match {
+                    case Some(targetValue) =>
+                      val pattern = "\\d+:\\d"
+                      if (targetValue.matches(pattern)) {


scala nit: you can rewrite this either if/else and nested clauses using case matching on regex.

hm.. can you show a example? my codes like below

if (targetValue.matches(pattern)) { val invokerArray = targetValue.split(":") val beginIndex = invokerArray(0).toInt val finishIndex = invokerArray(1).toInt if (finishIndex < beginIndex) { logging.info(this, "finishIndex can't be less than beginIndex") complete(StatusCodes.BadRequest) } else { val targetInvokers = (beginIndex to finishIndex).toList loadBalancer.sendRuntimeToInvokers(runtime, Some(targetInvokers)) logging.info(this, "config runtime request is already sent to target invokers") complete(StatusCodes.BadRequest) } } else { logging.info(this, "limit value can't match [beginIndex:finishIndex]") complete(StatusCodes.BadRequest) }

rabbah · 2020-01-26T16:01:24Z

core/invoker/src/main/scala/org/apache/openwhisk/core/invoker/DefaultInvokerServer.scala

+
+  private val invokerCredentials = loadConfigOrThrow[InvokerCredentials](ConfigKeys.invokerCredentials)
+
+  override def routes(implicit transid: TransactionId): Route = {


how will new runtime images get pulled to the invoker nodes? are you assuming they get pulled either via stem-cell "run" or when an action actually runs in the future?

When send to reqeust to backend (e.g. curl -u xxx:xxx-X POST http://xxx:xxx:10001/config/runtime), it will pull new runtime image immediately and create prewarm container if the stem-cell > 0.

rabbah · 2020-01-26T16:01:24Z

tests/src/test/scala/common/WhiskProperties.java

@@ -258,10 +258,40 @@ public static int getControllerBasePort() {
        return Integer.parseInt(whiskProperties.getProperty("controller.host.basePort"));
    }

+    public static String getControllerProtocol() {


i dont think any of these changes are necessary if you read configs through pureconfig.

Yes, already deleted.

rabbah · 2020-01-26T16:01:24Z

tests/src/test/scala/org/apache/openwhisk/operation/RuntimeConfiguration.scala

+        response.status shouldBe StatusCodes.OK
+      }
+
+    Thread.sleep(5.seconds.toMillis)


Just make previous's request handled successfully, because previous is a ansyc operation.

ningyougang · 2020-02-27T09:50:37Z

@rabbah

The limiting of a runtime to specific invokers is related to how invokers are partitioned for black box vs non-black box invokers (based on a % of active invokers in the system). I suggest splitting out the "limit" feature into a separate PR and having a mechanism for specifying distributions of runtimes to invokers and letting the load balancer calculate the partition.

hm.. this patch just reconfigure the runtime.json(recreate prewarm containers according to passed runtime string), i don't understand what's mean I suggest splitting out the "limit" feature into a separate PR

That isn't powerful enough to when it is desirable to route to a specific invoker - is that a desired feature (vs limiting to a range of invokers active in the system)?

Yes, it is not powerful, so i deleted route to a specific invoker because already have limiting to a range of invokers

ningyougang · 2020-02-27T09:53:40Z

core/controller/src/main/scala/org/apache/openwhisk/core/controller/Controller.scala

+                    case None =>
+                      loadBalancer.sendRuntimeToInvokers(runtime, None)
+                      logging.info(this, "config runtime request is already sent to all managed invokers")
+                      complete(StatusCodes.Accepted)


I changed to complete(StatusCodes.Accepted), because it is a async operation.

ningyougang · 2020-02-27T09:56:50Z

core/invoker/src/main/scala/org/apache/openwhisk/core/containerpool/ContainerPool.scala

@@ -279,6 +285,23 @@ class ContainerPool(childFactory: ActorRefFactory => ActorRef,
    case RescheduleJob =>
      freePool = freePool - sender()
      busyPool = busyPool - sender()
+    case prewarmConfigList: PreWarmConfigList =>
+      laststPrewarmConfig = prewarmConfigList.list


Add a inner variable laststPrewarmConfig, just make it pointed to the passed config runtime.
Because in this patch: #4698
when received ContainerRemoved(may be prewarm container crashed), it will backfill the prewarm.

ningyougang · 2020-03-09T01:07:51Z

Rebased.

ningyougang · 2020-06-08T06:27:45Z

Rebased

ningyougang · 2020-06-08T09:27:37Z

core/invoker/src/main/scala/org/apache/openwhisk/core/containerpool/ContainerPool.scala

+            passedConfig.exec.kind == config.exec.kind && passedConfig.memoryLimit == config.memoryLimit)
+          .getOrElse(config)
+      }
+      latestPrewarmConfig = newPrewarmConfig


Support just change specify runtime config as well, e.g.
Let's assume runtimes.json include nodejs:12, python:2, swift4.1, we can change nodejs:12 runtime only via just pass nodejs:12 runtime info, for other runtimes, just use previous runtime info directly.

codecov-commenter · 2020-06-08T10:03:11Z

Codecov Report

Merging #4790 into master will decrease coverage by 6.56%.
The diff coverage is 50.00%.

@@            Coverage Diff             @@
##           master    #4790      +/-   ##
==========================================
- Coverage   83.56%   76.99%   -6.57%     
==========================================
  Files         201      202       +1     
  Lines        9515     9613      +98     
  Branches      400      415      +15     
==========================================
- Hits         7951     7402     -549     
- Misses       1564     2211     +647

Impacted Files	Coverage Δ
...che/openwhisk/core/loadBalancer/LoadBalancer.scala	`76.92% <0.00%> (-6.42%)`	⬇️
...e/loadBalancer/ShardingContainerPoolBalancer.scala	`77.71% <0.00%> (-3.08%)`	⬇️
...la/org/apache/openwhisk/core/invoker/Invoker.scala	`71.66% <ø> (-0.47%)`	⬇️
.../apache/openwhisk/core/controller/Controller.scala	`45.11% <10.00%> (-9.70%)`	⬇️
.../org/apache/openwhisk/core/connector/Message.scala	`76.11% <37.50%> (-4.04%)`	⬇️
...pache/openwhisk/core/invoker/InvokerReactive.scala	`77.86% <64.40%> (-1.79%)`	⬇️
...e/openwhisk/core/containerpool/ContainerPool.scala	`94.25% <66.66%> (-3.62%)`	⬇️
.../openwhisk/core/invoker/DefaultInvokerServer.scala	`75.00% <75.00%> (ø)`
.../scala/org/apache/openwhisk/core/WhiskConfig.scala	`95.00% <100.00%> (+0.03%)`	⬆️
...rg/apache/openwhisk/core/entity/ExecManifest.scala	`92.15% <100.00%> (+0.32%)`	⬆️
... and 22 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 395149d...4157ebc. Read the comment docs.

ningyougang reviewed Jan 7, 2020

View changes

style95 reviewed Jan 7, 2020

View changes

ningyougang closed this Jan 13, 2020

ningyougang reopened this Jan 13, 2020

ningyougang force-pushed the ningyougang:config_runtime branch 3 times, most recently from 5be2eab to 8573912 Jan 13, 2020

ningyougang reviewed Jan 13, 2020

View changes

ningyougang force-pushed the ningyougang:config_runtime branch 2 times, most recently from 556b5bd to 1ec0720 Jan 14, 2020

ningyougang changed the title ~~[WIP]Config runtime~~ Config runtime Jan 14, 2020

rabbah reviewed Jan 26, 2020

View changes

rabbah changed the title ~~Config runtime~~ Add administrative interfafe to invoker and controller to reconfigure runtimes Jan 26, 2020

rabbah changed the title ~~Add administrative interfafe to invoker and controller to reconfigure runtimes~~ Add administrative interface to invoker and controller to reconfigure runtimes Jan 26, 2020

ningyougang force-pushed the ningyougang:config_runtime branch 3 times, most recently from 81caa3f to 3e0603d Feb 27, 2020

ningyougang reviewed Feb 27, 2020

View changes

ningyougang force-pushed the ningyougang:config_runtime branch 6 times, most recently from be5a6e5 to 6750ea9 Feb 28, 2020

ningyougang force-pushed the ningyougang:config_runtime branch 2 times, most recently from fc63c07 to 12201bb Feb 29, 2020

ningyougang force-pushed the ningyougang:config_runtime branch from 12201bb to c245908 Mar 9, 2020

ningyougang force-pushed the ningyougang:config_runtime branch 4 times, most recently from 6733631 to 30a3e64 Mar 23, 2020

ningyougang mentioned this pull request Mar 31, 2020

Change prewarm container #4225

Closed

6 of 19 tasks complete

ningyougang force-pushed the ningyougang:config_runtime branch from 30a3e64 to 842baa1 Jun 8, 2020

ningyougang closed this Jun 8, 2020

ningyougang reopened this Jun 8, 2020

ningyougang reviewed Jun 8, 2020

View changes

ningyougang mentioned this pull request Jun 9, 2020

take prewarmed container's memory as used memory #4911

Open

3 of 21 tasks complete

ningyougang force-pushed the ningyougang:config_runtime branch from 4157ebc to a5c2986 Sep 7, 2020

Config runtime

Loading status checks…

214f8e4

ningyougang force-pushed the ningyougang:config_runtime branch from a5c2986 to 214f8e4 Sep 8, 2020

apache / openwhisk

Add administrative interface to invoker and controller to reconfigure runtimes #4790

Add administrative interface to invoker and controller to reconfigure runtimes #4790

ningyougang commented Jan 7, 2020 •

edited

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

codecov-io commented Jan 13, 2020 •

edited

This comment has been minimized.

ningyougang commented Jan 14, 2020

rabbah left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ningyougang commented Feb 27, 2020 •

edited

This comment has been minimized.

This comment has been minimized.

ningyougang commented Mar 9, 2020

ningyougang commented Jun 8, 2020

This comment has been minimized.

codecov-commenter commented Jun 8, 2020 •

edited


		private val invokerCredentials = loadConfigOrThrow[InvokerCredentials](ConfigKeys.invokerCredentials)

		override def routes(implicit transid: TransactionId): Route = {

apache / openwhisk

Join GitHub today

GitHub is where the world builds software

Add administrative interface to invoker and controller to reconfigure runtimes #4790

Add administrative interface to invoker and controller to reconfigure runtimes #4790

Conversation

ningyougang commented Jan 7, 2020 • edited

Description

Related issue and scope

My changes affect the following components

Types of changes

Checklist:

This comment has been minimized.

ningyougang Jan 7, 2020 Author Contributor

This comment has been minimized.

ningyougang Jan 7, 2020 • edited Author Contributor

This comment has been minimized.

style95 Jan 7, 2020 Member

This comment has been minimized.

ningyougang Jan 13, 2020 Author Contributor

This comment has been minimized.

style95 Jan 7, 2020 Member

This comment has been minimized.

ningyougang Jan 13, 2020 Author Contributor

codecov-io commented Jan 13, 2020 • edited

Codecov Report

This comment has been minimized.

ningyougang Jan 13, 2020 Author Contributor

ningyougang commented Jan 14, 2020

rabbah left a comment

This comment has been minimized.

rabbah Jan 26, 2020 Member

This comment has been minimized.

ningyougang Feb 27, 2020 Author Contributor

This comment has been minimized.

rabbah Jan 26, 2020 Member

This comment has been minimized.

ningyougang Feb 27, 2020 Author Contributor

This comment has been minimized.

rabbah Jan 26, 2020 Member

This comment has been minimized.

ningyougang Feb 27, 2020 Author Contributor

This comment has been minimized.

rabbah Jan 26, 2020 Member

This comment has been minimized.

ningyougang Feb 27, 2020 Author Contributor

This comment has been minimized.

rabbah Jan 26, 2020 Member

This comment has been minimized.

ningyougang Feb 27, 2020 Author Contributor

This comment has been minimized.

rabbah Jan 26, 2020 Member

This comment has been minimized.

ningyougang Feb 27, 2020 • edited Author Contributor

This comment has been minimized.

rabbah Jan 26, 2020 Member

This comment has been minimized.

ningyougang Feb 27, 2020 • edited Author Contributor

This comment has been minimized.

rabbah Jan 26, 2020 Member

This comment has been minimized.

ningyougang Feb 27, 2020 Author Contributor

This comment has been minimized.

rabbah Jan 26, 2020 Member

This comment has been minimized.

ningyougang Feb 27, 2020 Author Contributor

ningyougang commented Feb 27, 2020 • edited

This comment has been minimized.

ningyougang Feb 27, 2020 Author Contributor

This comment has been minimized.

ningyougang Feb 27, 2020 Author Contributor

ningyougang commented Mar 9, 2020

ningyougang commented Jun 8, 2020

This comment has been minimized.

ningyougang Jun 8, 2020 • edited Author Contributor

codecov-commenter commented Jun 8, 2020 • edited

Codecov Report

Essential cookies

Always active

Analytics cookies

ningyougang commented Jan 7, 2020 •

edited

ningyougang Jan 7, 2020
Author Contributor

ningyougang Jan 7, 2020 •

edited

Author Contributor

style95 Jan 7, 2020
Member

ningyougang Jan 13, 2020
Author Contributor

style95 Jan 7, 2020
Member

ningyougang Jan 13, 2020
Author Contributor

codecov-io commented Jan 13, 2020 •

edited

ningyougang Jan 13, 2020
Author Contributor

rabbah Jan 26, 2020
Member

ningyougang Feb 27, 2020
Author Contributor

rabbah Jan 26, 2020
Member

ningyougang Feb 27, 2020
Author Contributor

rabbah Jan 26, 2020
Member

ningyougang Feb 27, 2020
Author Contributor

rabbah Jan 26, 2020
Member

ningyougang Feb 27, 2020
Author Contributor

rabbah Jan 26, 2020
Member

ningyougang Feb 27, 2020
Author Contributor

rabbah Jan 26, 2020
Member

ningyougang Feb 27, 2020 •

edited

Author Contributor

rabbah Jan 26, 2020
Member

ningyougang Feb 27, 2020 •

edited

Author Contributor

rabbah Jan 26, 2020
Member

ningyougang Feb 27, 2020
Author Contributor

rabbah Jan 26, 2020
Member

ningyougang Feb 27, 2020
Author Contributor

ningyougang commented Feb 27, 2020 •

edited

ningyougang Feb 27, 2020
Author Contributor

ningyougang Feb 27, 2020
Author Contributor

ningyougang Jun 8, 2020 •

edited

Author Contributor

codecov-commenter commented Jun 8, 2020 •

edited