apache / pulsar

fix #8533 , modified only issue-related method, may need to change all calls in this section to asynchronous later.

---

Master Issue: #8676

*Motivation*

Show REST API examples in the website.

*Modifications*

Add website related changes for package management REST APIs.

*ScreenShot*

![Screen Shot 2020-12-10 at 5 25 18 PM](https://user-images.githubusercontent.com/24502569/101753478-a42da300-3b0d-11eb-9aa9-ec07f0354ee8.png)

…oid deadlocks (#8877)

### Motivation

Some of our broker servers experienced what appears to be a deadlock. The following is the thread dump at that time.

[threaddump.txt.zip](https://github.com/apache/pulsar/files/5665572/threaddump.txt.zip)

The thread "ForkJoinPool.commonPool-worker-120" was locking an instance of `ManagedLedgerImpl`. And this thread seemed to be waiting for `subscriptions`, which is an instance of `ConcurrentOpenHashMap`, to be unlocked. Many other threads were blocked because the lock on the `ManagedLedgerImpl` instance was not released.

```
"ForkJoinPool.commonPool-worker-120" #903 daemon prio=5 os_prio=0 tid=0x00007f9aa0010000 nid=0x12b59 waiting on condition [0x00007f9528cc3000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007fa20b3e5eb0> (a org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section)
        at java.util.concurrent.locks.StampedLock.acquireWrite(StampedLock.java:1119)
        at java.util.concurrent.locks.StampedLock.writeLock(StampedLock.java:354)
        at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.put(ConcurrentOpenHashMap.java:245)
        at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.computeIfAbsent(ConcurrentOpenHashMap.java:129)
        at org.apache.pulsar.broker.service.persistent.PersistentTopic$2.openCursorComplete(PersistentTopic.java:650)
        at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.asyncOpenCursor(ManagedLedgerImpl.java:720)
        - locked <0x00007fa20512f968> (a org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl)
        at org.apache.pulsar.broker.service.persistent.PersistentTopic.getDurableSubscription(PersistentTopic.java:643)
        at org.apache.pulsar.broker.service.persistent.PersistentTopic.subscribe(PersistentTopic.java:590)
        at org.apache.pulsar.broker.service.ServerCnx.lambda$null$10(ServerCnx.java:699)
        at org.apache.pulsar.broker.service.ServerCnx$$Lambda$476/1880414247.apply(Unknown Source)
        at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
        at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
        at org.apache.pulsar.broker.service.ServerCnx.lambda$null$13(ServerCnx.java:682)
        at org.apache.pulsar.broker.service.ServerCnx$$Lambda$475/707554512.apply(Unknown Source)
        at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
        at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
        at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
        at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:943)
        at java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:457)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
```

The thread that locked `subscriptions` seems to be "pulsar-msg-expiry-monitor-24-1".
```
"pulsar-msg-expiry-monitor-24-1" #304 prio=5 os_prio=0 tid=0x00007f99602dd000 nid=0x12036 waiting on condition [0x00007f998d47c000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007fca4361dfb0> (a java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl.getNthEntry(ManagedCursorImpl.java:537)
        at org.apache.pulsar.broker.service.persistent.PersistentTopic.isOldestMessageExpired(PersistentTopic.java:1820)
        at org.apache.pulsar.broker.service.persistent.PersistentSubscription.expireMessages(PersistentSubscription.java:901)
        at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$checkMessageExpiry$36(PersistentTopic.java:1102)
        at org.apache.pulsar.broker.service.persistent.PersistentTopic$$Lambda$1011/2104832020.accept(Unknown Source)
        at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:385)
        at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159)
        at org.apache.pulsar.broker.service.persistent.PersistentTopic.checkMessageExpiry(PersistentTopic.java:1102)
        at org.apache.pulsar.broker.service.BrokerService$$Lambda$1009/2005752676.accept(Unknown Source)
        at org.apache.pulsar.broker.service.BrokerService.lambda$forEachTopic$32(BrokerService.java:951)
        at org.apache.pulsar.broker.service.BrokerService$$Lambda$779/1852910990.accept(Unknown Source)
        at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:385)
        at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159)
        at org.apache.pulsar.broker.service.BrokerService.forEachTopic(BrokerService.java:948)
        at org.apache.pulsar.broker.service.BrokerService.checkMessageExpiry(BrokerService.java:925)
        at org.apache.pulsar.broker.service.BrokerService$$Lambda$108/203149502.run(Unknown Source)
        at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32)
        at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)
```

I can't understand why "pulsar-msg-expiry-monitor-24-1" was stuck. However, it seems that this deadlock can be avoided if `subscriptions` is not locked when checking for message expiration, so I created this PR. If anyone can explain why "pulsar-msg-expiry-monitor-24-1" was stuck, please let me know.

### Modifications

When expiring messages for each subscription, copy the values of `subscriptions` as `List` and execute `forEach()` for that `List` instance.

Signed-off-by: xiaolong.ran <rxl@apache.org>

Fixes #6527 

### Motivation

We already support `key_shared` mode on Pulsar Functions in #7647, However, this option is not exposed in the command line parameters, so users currently cannot use this function.

### Modifications

In this pull request, we expose this cmd flag of `--retain-key-ordering` and add the docs for this.

### Verifying this change

The original test can cover this option, so, in this change, no relevant code is added

…IntelliJ (#8892)

### Motivation

The current instructions to setup Pulsar development in IntelliJ could be improved. 

### Modifications

Update README.md
* improved instructions for Pulsar development in IntelliJ
* update the information that building Pulsar with newer than Java 8 has issues (see the changes for details)

…#8903)

Master Issue: #8843 

### Motivation

Enable spotbugs for the pulsar-io module. 

### Modifications

Enable soptbugs for pulsar-io/core, pulsar-io/common, pulsar-io/docs.

### Verifying this change

```
mvn clean install -pl pulsar-io/core
mvn clean install -pl pulsar-io/common
mvn clean install -pl pulsar-io/docs

```

…am Sinks (#8854)

## Changes:
- extract original Schema from the message read from the topic
- expose the schema into the PulsarRecord implementation
- add unit test
- in PulsarSink detect the fact that the source record is directly from a PulsarSource and drop the schema (there was already a check about the fact that the source record is a PulsarRecord, so this patch is not adding a new shortcircuit)
- little refactor in PulsarSink in order to make it clear that we are working with SinkRecord (casts were already present in the code, and it would not have worked without SinkRecord, so we are only tidying it up)

### Motivation
Pulsar sinks cannot access the original Schema of the Message they are consuming,
this is because currently PulsarSource does not publish the Schema information to downstream processing.

### Modifications
Try to retrieve Schema information from the Message and push it down to the Record.

### Verifying this change
- [X] Make sure that the change passes the CI checks.
- [X] This change added tests and can be verified as follows:
- added unit test
- tested manually with Pulsar Standalone 
- there are existing unit and integration tests that are touching this change (in fact I had to iterate a few times)

### Does this pull request potentially affect one of the following parts: yes
with this change Pulsar function users and especially Pulsar I/O Sinks will be able to inspect the original schema in case the processing starts from a Pulsar topic (PulsarSource)

### Motivation

Google's [Error Prone static code analysis tool](https://errorprone.info/) can detect typical programming mistakes (for more info see https://errorprone.info/). This PR adds configuration to Pulsar maven build so that Error Prone static code analysis
can be run for the Pulsar code base. This additional configuration is not active by default and requires activating `errorprone` and `errorprone-jdk8` or `errorprone-jdk11` maven profiles in addition to `main` or `core-modules` profile for running the analysis.

### Modifications

- add a separate maven profile for running errorprone analysis
  it is not active by default

- Add top-level `lombok.config file` for configuring Lombok.
  This file doesn't contain any other settings that
  marks it as the top most config file with `config.stopBubbling = true`
  (see https://projectlombok.org/features/configuration for reference)

- Before running the Error Prone analysis, it's necessary to
  manually configure Lombok to add `@javax.annotation.Generated("lombok")`
  annotation on all fields, methods, and types that are generated.
  This makes Error Prone skip the analysis for Lombok generated code.
  - this setting cannot be enabled by default since JDK9+ doesn't include
    the `javax.annotation.Generated` annotation by default and adding the
    required dependency will change the classpath dependency resolution results.
  - There is an open issue to support lombok.Generated annotion in
    ErrorProne, google/error-prone#1863

- example usage:
  Configure Lombok to add Generated annotations
  `echo "lombok.addJavaxGeneratedAnnotation = true" >> lombok.config`
  then,
  on JDK8
  `mvn -Perrorprone,errorprone-jdk8,core-modules compile`
  on JDK11+
  `mvn -Perrorprone,errorprone-jdk11,core-modules compile`

  - usability is better when used together with IntelliJ
    since one can click on the error message to navigate
    to the code location

- also add configuration for Error Prone SLF4J plugin
  https://github.com/KengoTODA/errorprone-slf4j
  which helps detect misusage of SLF4J API

…ndency with log4j-1.2-api (#8880)

- import Log4j2's BOM into Pulsar's dependency management instead of
  defining versions separately for each Log4j2 library

- distribution/server/pom.xml included log4j:log4j dependency.
  - replace with org.apache.logging.log4j:log4j-1.2-api to
    keep supporting the legacy logging api in the case something
    depends on it

### Motivation

This is an update after #8888, where `functions` should jump to `/functions-rest-api`,  `source` should jump to `/source-rest-api`, `sink` should jump to `/sink-rest-api`

### Modifications

* Update route jump
* Update markdown content render

---

Master Issue: #8676

*Motivation*

Add integration tests for testing the whole package management service.

*Modifications*

- Add integration tests
- Fix some issue when startup the package management service

…nd batch-data-generator (#8889)

Thanks for your contribution

---

*Motivation*

We have some CIs are not using the latest image for running integration
tests. We should always build the latest image using the current code
to run the integration tests.

*Modifications*

- Add build docker steps in the ci files

Master Issue: #8853

### Motivation

This pr adds a `[[pulsar.version_number]]` field to restapi in documentation after 2.3.0, and fixes the build script.


### Modifications

* Add `pulsar.version_number`
* Fixed replace script

### Verifying this change

- [ ] Make sure that the change passes the CI checks.

After building the script:

![image](https://user-images.githubusercontent.com/1907867/101728310-31163380-3af1-11eb-9157-3045d3282025.png)

![image](https://user-images.githubusercontent.com/1907867/101728987-59eaf880-3af2-11eb-92f0-5f66f411af7f.png)

#8883)

Describe the bug
It looks like GenericJsonReader is not handling correctly null values

Expected behavior
The null value is not converted to a string, but it is still a null value

Changes
- Handle correctly null values
- add test case

Additional context
The problem affects Pulsar Functions/Pulsar IO

Fixes #8882

Fix issue: streamnative#1768

This PR contains code style changes only, it will be the final patch to make module `pulsar-broker` conform checkstyle.

…admin API namespace doc (#8760)

Fixes streamnative#1639



### Motivation
To specify how soon after message acknowledgement can we expect the message to be deleted - is it immediate or is there some asynchronous cleanup process.



### Modifications

1: Add "delete data from namespaces" for master release. Once the PR is approved, will sync update to previous releases.
2: Reorg the Admin API for namespace doc as some section titles are missed.

### Motivation

In out build environment (MagNews.com) we are upgrading Swagger to 1.6.2, as it is also used by Pulsar Builder system we would like to upgrade Swagger here as well.
Even if Swagger is used for documentation, swagger-core is a runtime dependency so it is better to guarantee that Pulsar is working well with latest version of Swagger 1.X.

I am also upgrading the Maven Swagger Plugin to 3.1.7, the 3.1.8 is buggy and it fails the build with NPE (the same problem occurs even in MagNews build, so it is not related to Pulsar itself)

### Modifications
- Upgrade Swagger version to 1.6.2
- Upgrade Maven Plugin to 3.1.7

### Motivation
Generate full amount of documentation for version 2.7.0, release manager run the `yarn run version <release-version>`  command to generate the full amount of documentation.


### Modifications

* Add a script `docusaurus-version.js`
* Update command `yarn run version`
* Generate full docs for pulsar 2.7.0

---
    
Master Issue: #8676
    
*Motivation*
    
Add REST API resources for package management service and
allow the package management service to handle HTTP requests
    
*Modifications*
    
- Add REST APIs for package management service

…ackage (#8861)

Master Issue: #8676

*Motivation*

Currently, the package management bookkeeper storage jar is not in the distribution
package so the service can not find the storage class to start up.

*Modifications*

- Add pulsar-package-bookkeeper-storage.jar to the distribution module

---

*Motivation*

Disable the cherry-pick bot. Because of the cherry-pick bot
will change the author of the PR and maybe need another way
to do the cherry-pick automatically.

* Current unit test group for ci is scattered and error prone.
* This PR unifies all group declarations in single file for easier use and simpler management.

### Motivation
The "Manage Schema" content is merged into "Pulsar Schema" section, yet there are some links to it, and the blank content is still in the sidebar file. The changes was applied since 2.5.0.

### Modifications
- Resolve the link issues.
- Fix the sidebar since 2.5.0.

…#8862)

Main issue: streamnative#1768
This PR changes code style only.

Master Issue: #8843

### Motivation

Enable spotbugs for the pulsar-io module. 

### Modifications

Enable soptbugs for pulsar-io-netty

### Verifying this change

```
mvn clean install -pl pulsar-io/netty
```

Master Issue: #8787 

### Motivation

Currently, the reader subscription name can only be generated internally randomly in the C++ client.
Java client part is at #8801 

### Modifications

Add a setter for the reader's internal subscription name.

### Verifying this change

This change is already covered by existing tests, such as *testSubscriptionNameSetting*, *testSetSubscriptionNameAndPrefix* and *testMultiSameSubscriptionNameReaderShouldFail*.

---

Master Issue: #8676

*Motivation*

Add commands for package management service in pulsar-admin tool.

*Modifications*

- Add commands for package management service

### Motivation
Sync changes from #8762

### Motivation

BlockingQueue has race condition that can cause threads waiting forever in multithreading environment. ProducerImpl uses BlockingQueue as pendingMessagesQueue_ and can be blocked forever at it. This PR fixes race condition in BlockingQueue.

#### Race condition details
https://github.com/apache/pulsar/blob/91e2f832178d9ffd5d78161145d895910296c2d9/pulsar-client-cpp/lib/BlockingQueue.h#L172-L185
Use BlockingQueue::Pop as example, its procedure is:
1. lock
2. check wasFull and then change queue state
3. unlock
4. if wasFull, notify one thread waiting at queueFullCondition

Race condition sequence:
1. queue is full and there are multiple threads waiting on queueFullCondition
2. thread A call Pop, lock, wasFull is true, unlock -> queue has one free space
3. thread B call Pop, lock, wasFull is false, unlock -> queue has two free spaces
4. thread A notify one thread waiting at queueFullCondition
5. queue is no loger full again
6. result: except one thread is notified by A, other threads waiting on queueFullCondition are waiting forever

### Modifications

* Use notify_all instead of notify_one to notify threads waiting on condition variables
  Reason: Currently only notify threads when queue is full or empty. After unlock, other threads may change queue state, so thread to notify condition can not determine how queue state changed and should use notify_all in case of more then one  change occured.

### Verifying this change

  - Add a test case BlockingQueueTest.testPushPopRace to test concurrent push and pop

Add provided `jclouds-blobstore` for module `tiered-storage`.

A provided dependency will not change the build result of a maven project, but it will provide information to make IDEs easier to parse and analyze the project dependencies rather than a highly customized shaded dependency.

### Motivation
When the value of "/bookies" in ZooKeeper is changed, Brokers are notified of the change.
After the notification to Brokers, only one of the two BookKeeper clients that Brokers have seems to reflect the change.

**Two BookKeeper clients that Brokers have**
- https://github.com/apache/pulsar/blob/ac0c6e41f0ebe3c900bb31e41c8d40b3f60b19df/pulsar-broker/src/main/java/org/apache/pulsar/broker/ManagedLedgerClientFactory.java#L81
- https://github.com/apache/pulsar/blob/102fa9de03509b86e47f58ab8e1c0dde2095da3b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/BookkeeperSchemaStorage.java#L105

The client of `BookkeeperSchemaStorage` don't seems to reflect the change.
The cause is that `ZkBookieRackAffinityMapping#onUpdate` don't run.

I confirmed that I change ZkBookieRackAffinityMapping instances to use same `ZooKeeperDataCache` and `ZkBookieRackAffinityMapping#onUpdate` works.

### Modification
- Set the argument of `ZkBookieRackAffinityMapping#setConf` to `bookieMappingCache`
- Move `updateRacksWithHost()`

Signed-off-by: xiaolong.ran <rxl@apache.org>

### Motivation

* Enable spotbugs for the pulsar-client-tools-test module.

### Modifications

* Enable spotbugs for the pulsar-client-tools-test module.

### Verifying this change

```
mvn clean install -pl pulsar-client-tools-test
```

…m CheckStyle (#8848)

* add provided jclouds

Signed-off-by: Renkai <gaelookair@gmail.com>

* make broker-style gradually conform checkstyle

Signed-off-by: Renkai <gaelookair@gmail.com>

* make broker-style gradually conform checkstyle

Signed-off-by: Renkai <gaelookair@gmail.com>

* make broker-style gradually conform checkstyle

Signed-off-by: Renkai <gaelookair@gmail.com>

* make broker-style gradually conform checkstyle

Signed-off-by: Renkai <gaelookair@gmail.com>