Enterprise Server 3.1 release notes

CRITICAL: A remote code execution vulnerability in the Log4j library, identified as CVE-2021-44228, affected all versions of GitHub Enterprise Server prior to 3.3.1. The Log4j library is used in an open source service running on the GitHub Enterprise Server instance. This vulnerability was fixed in GitHub Enterprise Server versions 3.0.22, 3.1.14, 3.2.6, and 3.3.1. For more information, please see this post on the GitHub Blog.

Support bundles could include sensitive files if they met a specific set of conditions.

Running ghe-config-apply could sometimes fail because of permission issues in /data/user/tmp/pages.

A misconfiguration in the Management Console caused scheduling errors.

Docker would hold log files open after a log rotation.

GraphQL requests did not set the GITHUB_USER_IP variable in pre-receive hook environments.

Clarifies explanation of Actions path-style in documentation.

Updates support contact URLs to use the current support site, support.github.com.

包已更新到最新的安全版本。

Running ghe-repl-start or ghe-repl-status would sometimes return errors connecting to the database when GitHub Actions was enabled.

Pre-receive hooks would fail due to undefined PATH.

Running ghe-repl-setup would return an error: cannot create directory /data/user/elasticsearch: File exists if the instance had previously been configured as a replica.

After setting up a high availability replica, ghe-repl-status included an error in the output: unexpected unclosed action in command.

In large cluster environments, the authentication backend could be unavailable on a subset of frontend nodes.

Some critical services may not have been available on backend nodes in GHES Cluster.

An additional outer layer of gzip compression when creating a cluster support bundle with ghe-cluster-suport-bundle is now turned off by default. This outer compression can optionally be applied with the ghe-cluster-suport-bundle -c command line option.

We have added extra text to the admin console to remind users about the mobile apps' data collection for experience improvement purposes.

The GitHub Connect data connection record now includes a list of enabled GitHub Connect features. [Updated 2021-12-09]

A path traversal vulnerability was identified in GitHub Pages builds on GitHub Enterprise Server that could allow an attacker to read system files. To exploit this vulnerability, an attacker needed permission to create and build a GitHub Pages site on the GitHub Enterprise Server instance. This vulnerability affected all versions of GitHub Enterprise Server prior to 3.3, and was fixed in versions 3.0.19, 3.1.11, and 3.2.3. This vulnerability was reported through the GitHub Bug Bounty program and has been assigned CVE-2021-22870.

包已更新到最新的安全版本。

Some Git operations failed after upgrading a GitHub Enterprise Server 3.x cluster because of the HAProxy configuration.

Unicorn worker counts might have been set incorrectly in clustering mode.

Resqued worker counts might have been set incorrectly in clustering mode.

If Ubuntu's Uncomplicated Firewall (UFW) status was inactive, a client could not clearly see it in the logs.

Upgrading from GitHub Enterprise Server 2.x to 3.x failed when there were UTF8 characters in an LDAP configuration.

Some pages and Git-related background jobs might not run in cluster mode with certain cluster configurations.

When a new tag was created, the push webhook payload did not display a correct head_commit object. Now, when a new tag is created, the push webhook payload now always includes a head_commit object that contains the data of the commit that the new tag points to. As a result, the head_commit object will always contain the commit data of the payload's after commit.

The enterprise audit log page would not display audit events for 秘密扫描.

There was an insufficient job timeout for replica repairs.

Users were not warned about potentially dangerous bidirectional unicode characters when viewing files. For more information, see "Warning about bidirectional Unicode text" in GitHub 博客.

Hookshot Go sent distribution type metrics that Collectd could not handle, which caused a ballooning of parsing errors.

Kafka configuration improvements have been added. When deleting repositories, package files are now immediately deleted from storage account to free up space. DestroyDeletedPackageVersionsJob now deletes package files from storage account for stale packages along with metadata records.

It was possible for cleartext passwords to end up in certain log files.

Several known weak SSH public keys have been added to the deny list and can no longer be registered. In addition, versions of GitKraken known to generate weak SSH keys (7.6.x, 7.7.x and 8.0.0) have been blocked from registering new public keys.

包已更新到最新的安全版本。

Restore might fail for enterprise server in clustering mode if orchestrator isnt healthily.

Several parts of the application were unusable for users who are owners of many organizations.

Fixed a link to https://docs.github.com.

Browsing and job performance optimizations for repositories with many refs.

包已更新到最新的安全版本。

Custom pre-receive hooks could have failed due to too restrictive virtual memory or CPU time limits.

Attempting to wipe all existing configuration settings with ghe-cleanup-settings failed to restart the Management Console service.

During replication teardown via ghe-repl-teardown Memcached failed to be restarted.

During periods of high load, users would receive HTTP 503 status codes when upstream services failed internal healthchecks.

With Actions configured, MSSQL replication would fail after restoring from a GitHub Enterprise Backup Utilities snapshot.

An erroneous jq error message may have been displayed when running ghe-config-apply.

Pre-receive hook environments were forbidden from calling the cat command via BusyBox on Alpine.

The external database password was logged in plaintext.

Failing over from a primary Cluster datacenter to a secondary Cluster datacenter succeeds, but then failing back over to the original primary Cluster datacenter failed to promote Elasticsearch indicies.

The "Import teams" button on the Teams page for an Organization returned an HTTP 404.

In some cases, GitHub Enterprise Administrators attempting to view the Dormant users page received 502 Bad Gateway or 504 Gateway Timeout response.

Performance was negatively impacted in certain high load situations as a result of the increased number of SynchronizePullRequestJob jobs.

More effectively delete Webhook logs that fall out of the Webhook log retention window.

HIGH: A path traversal vulnerability was identified in GitHub Enterprise Server that could be exploited when building a GitHub Pages site. User-controlled configuration options used by GitHub Pages were not sufficiently restricted and made it possible to read files on the GitHub Enterprise Server instance. To exploit this vulnerability, an attacker would need permission to create and build a GitHub Pages site on the GitHub Enterprise Server instance. This vulnerability affected all versions of GitHub Enterprise Server prior to 3.1.8 and was fixed in 3.1.8, 3.0.16, and 2.22.22. This is the result of an incomplete fix for CVE-2021-22867. This vulnerability was reported via the GitHub Bug Bounty program and has been assigned CVE-2021-22868.

MEDIUM: An improper access control vulnerability in GitHub Enterprise Server allowed a workflow job to execute in a self-hosted runner group it should not have had access to. This affects customers using self-hosted runner groups for access control. A repository with access to one enterprise runner group could access all of the enterprise runner groups within the organization because of improper authentication checks during the request. This could cause code to be run unintentionally by the incorrect runner group. This vulnerability affected GitHub Enterprise Server versions from 3.0.0 to 3.0.15 and 3.1.0 to 3.1.7 and was fixed in 3.0.16 and 3.1.8 releases. It has been assigned CVE-2021-22869.

Resque worker counts were displayed incorrectly during maintenance mode.

Allocated memcached memory could be zero in clustering mode.

Non-empty binary files displayed an incorrect file type and size on the pull request "Files" tab.

Fixes GitHub Pages builds so they take into account the NO_PROXY setting of the appliance. This is relevant to appliances configured with an HTTP proxy only. (update 2021-09-30)

The GitHub Connect configuration of the source instance was always restored to new instances even when the --config option for ghe-restore was not used. This would lead to a conflict with the GitHub Connect connection and license synchronization if both the source and destination instances were online at the same time. The fix also requires updating backup-utils to 3.2.0 or higher. [updated: 2021-11-18]

包已更新到最新的安全版本。

Attempting to tear down a newly-added replica node by specifying its UUID with ghe-repl-teardown would fail without reporting an error if replication was not started.

GitHub Pages builds were being passed through an external proxy if there was one configured.

Custom pre-receive hooks that created sub-processes would lack a PATH variable in their environment, resulting in "No such file or directory" errors.

MySQL could failover during an upgrade if mysql-auto-failover was enabled.

包已更新到最新的安全版本。

Attaching very large images or animated GIFs to images or pull requests would fail.

Journald messages related to automatic updates (Adding h/m/s random time.) were logged to syslog.

Custom pre-receive hooks that created named pipes (FIFOs) would crash or hang, resulting in a timeout error.

Adding filters to the audit log advanced search page did not populate the query text box in real-time with the correct facet prefix and value.

Git hooks to the internal API that result in failing requests returned the exception undefined method body for "success":String (NoMethodError) instead of returning an explicit nil.

When an integration was removed, it was possible for an unrelated OAuth application or integration to also be removed.

When a mandatory message containing an emoji character was added, attempting to view or change the message would return a 500 Internal Server Error.

Adds triage and maintain to the list of permissions returned by the REST API.

Custom pre-receive hooks that used a bash subshell would return an error: No such file or directory.

When GitHub Actions is enabled without running regular scheduled backups the MSSQL Transaction Log could grow unbounded and can consume all available space on the appliance's Data Disk causing a possible outage.

Unnecessary database logging consumed a large amount of disk space on instances with heavy LFS usage.

Audit log entries for changes made to "Repository creation" organization settings were inaccurate.

Excessive logging of ActionController::UnknownFormat exceptions caused unnecessary disk usage.

LDAP group_dn values longer than 255 characters would result in errors being logged: Data truncated for column 'group_dn' at row 1.

Abuse rate limits are now called Secondary rate limits, since the behavior they limit is not always abusive.

包已更新到最新的安全版本。

The counts on packages pages were not being incremented when a package was downloaded.

ghe-config-apply would timeout, ask for a prompt or fail for a customer that had 秘密扫描 enabled, and had either disabled or never enabled GitHub Actions on their instance.

Log files were not reopened after rotation in some cases leading to high disk space usage on instances with high uptime.

Upgrade could fail from older version of GitHub Enterprise Server due to a missing job in GitHub Actions.

Custom pre-receive hooks could lead to an error like error: object directory /data/user/repositories/0/nw/12/34/56/7890/network.git/objects does not exist; check .git/objects/info/alternates.

Unauthenticated HTTP proxy for the pages containers build was not supported for any users that use HTTP proxies.

A significant number of 503 errors were logged every time a user visited a repository's /settings page if the dependency graph was not enabled.

Internal repositories were only returned when a user had affiliations with the repository through a team or through collaborator status, or queried with the ?type=internal parameter.

Failed background jobs had unlimited retries which could cause large queue depths.

A significant number of 503 errors were being created if the scheduled job to sync vulnerabilities with GitHub.com attempted to run when dependency graph was not enabled and content analysis was enabled.

When GitHub Actions is enabled without running regular scheduled backups, the MSSQL transaction log could grow unbounded and can consume all available space on the appliance's data disk, causing a possible outage.

If you have configured regularly scheduled MSSQL backups, no further actions is required. Otherwise, if you have GitHub Actions previously enabled, run the following commands after installing this patch.

ghe-actions-console -s Mps -c 'Update-Service -Force'
ghe-actions-console -s Token -c 'Update-Service -Force'
ghe-actions-console -s Actions -c 'Update-Service -Force'

The logs for babeld now include a cmd field for HTTP ref advertisement requests instead of only including it during the negotiation requests.

HIGH: A path traversal vulnerability was identified in GitHub Enterprise Server that could be exploited when building a GitHub Pages site. User-controlled configuration options used by GitHub Pages were not sufficiently restricted and made it possible to read files on the GitHub Enterprise Server instance. To exploit this vulnerability, an attacker would need permission to create and build a GitHub Pages site on the GitHub Enterprise Server instance. This vulnerability affected all versions of GitHub Enterprise Server prior to 3.1.3 and has been assigned CVE-2021-22867. This vulnerability was reported via the GitHub Bug Bounty program.

包已更新到最新的安全版本。

SAML expiration date variable was not configurable.

Application services would fail their health checks during config apply before they could enter a healthy state.

ghe-cluster-config-node-init would fail during cluster setup if HTTP proxy is enabled.

Pre-receive hooks could encounter an error Failed to resolve full path of the current executable due to /proc not being mounted on the container.

Collectd would not resolve the forwarding destination hostname after the initial startup.

The job that purged stale deleted repositories could fail to make progress if some of those repositories were protected from deletion by legal holds.

Background jobs were being queued to the spam queue which were not being processed.

The preferred merge method would be reset when retrying after a failed PR merge.

Git pushes could result in a 500 Internal Server Error during the user reconciliation process on instances using LDAP authentication mode.

After upgrading from 3.0.x to 3.1.x, in some cases GitHub Actions would fail with an error: An unexpected error occurred when executing this workflow.

Improved the efficiency of config apply by skipping IP allow firewall rules that had not changed, which saved significant time on large clusters.

包已更新到最新的安全版本。

管理控制台中可能会积累大量 gauge-dependency-graph-api-dispatch_dispatch 指标。

sshd 服务有时无法从 Google Cloud 平台上运行的实例开始。

旧的升级文件将持续在用户磁盘上，有时导致空间不足。

gh-migrator 显示了其日志输出的不正确路径。

如果导出文件包含不在存档中的团队的审查请求，则导出存档在导入拉取请求时将产生无提示的失败。

将 GHES 3.1 中的 GitHub Actions Runner 版本更新到 v2.278.0

包已更新到最新的安全版本。

SVN 1.7 及以上客户端在使用 svn co 和 svn export 命令时出现错误。

使用 ghe-repo <owner>/<reponame> 通过管理外壳访问仓库将被挂起。

升级后，用户在大量使用时会减少可用性，因为服务重启太频繁。 出现这种情况是因为 nomad 配置与内部服务器的配置之间超时不匹配。

在某些情况下，设置 GitHub Actions 后运行 ghe-repl-status 会产生错误，并且 ghe-actions-teardown 会失败。

ghe-dbconsole 会在某些情况下返回错误。

从 非 GitHub 源导入组织或仓库失败可能会产生 undefined method '[]' for nil:NilClass 错误。

使用 SAML 身份验证时，如果 GitHub 配置文件名称不匹配管理控制台中映射到“Full name（全名）”字段的属性值，GitHub 配置文件名称可能已无意中更改。

升级以前运行过 2.13 版本但未运行 2.14 版本的实例，会导致与 AddRepositoryIdToCheckRuns 数据转换相关的数据库迁移错误。

GraphQL API 的用户可以在 PullRequest 对象上查询公共字段 closingIssuesReferences 。此字段检索在相关拉取请求合并时自动关闭的议题。此方法还允许未来迁移此数据，作为更高保真度移徙过程的一部分。

Secret Scanning is now generally available on GitHub Enterprise Server 3.1+. Scan public and private repositories for committed credentials, find secrets, and notify the secret provider or admin the moment they are committed into a repository.

This release includes several improvements from the beta of Secret Scanning on GitHub Enterprise Server:

• Expanded our pattern coverage from 24 partners to 37
• Added an API and webhooks
• Added notifications for commit authors when they commit secrets
• Updated the index view to made it easy to triage secrets in bulk
• Reduced the false positive rate on many patterns

Administrators using GitHub Advanced Security can enable and configure GitHub Advanced Security secret scanning. You can review the updated minimum requirements for your platform before you turn on GitHub Advanced Security secret scanning.

This release includes several improvements to GitHub Advanced Security billing in GitHub Enterprise Server:

• GitHub Advanced Security customers can now view their active committer count and the remaining number of unused committer seats on their organization or enterprise account’s Billing page. If Advanced Security is purchased for an enterprise, administrators can also view the active committer seats which are being used by other organizations within their enterprise. For more information, see "About GitHub Advanced Security licensing" and "Viewing your GitHub Advanced Security usage."
• GitHub Advanced Security customers can now view their active committer count for any Advanced Security enabled repositories on their organization or enterprise account's Billing page. These changes help billing administrators track their usage against how many committer licenses they purchased. For more information see "Managing security and analysis settings for your organization."

This release includes improvements to Dependabot alerts in GitHub Enterprise Server:

• Users with Dependabot alerts enabled can see which of their repositories are impacted by a given vulnerability by navigating to its entry in the GitHub Advisory Database. This feature is available in public beta. For more information, see "Viewing and updating vulnerable dependencies in your repository."
• When a vulnerability is added to GitHub Advisory Database, you will no longer receive email and web notifications for Dependabot alerts on low and moderate severity vulnerabilities. These alerts are still accessible from the repository's Security tab. For more information, see "Viewing and updating vulnerable dependencies in your repository."
• You can now give people instructions on how to responsibly report security vulnerabilities in your project by adding a SECURITY.md file to your repository's root, docs, or .github folder. When someone creates an issue in your repository, they will see a link to your project's security policy. For more information, see "Adding a security policy to your repository."

GitHub Actions can now generate a visual graph of your workflow on every run. With workflow visualization, you can:

• View and understand complex workflows
• Track progress of workflows in real-time
• Troubleshoot runs quickly by easily accessing logs and jobs metadata
• Monitor progress of deployment jobs and easily access deployment targets

For more information, see "Using the visualization graph."

OAuth 2.0 Device Authorization Grant allows any CLI client or developer tool to authenticate using a secondary system with a browser.

Administrators using OAuth Apps and GitHub Apps can enable and configure OAuth 2.0 Device Authorization Flow, in addition to the existing Web Application Flow. You can review the updated minimum requirements for your platform before you enable OAuth 2.0 Device Authorization Flow.

With auto-merge, pull requests can be set to merge automatically when all merge requirements have been satisfied. This saves users from needing to constantly check the state of their pull requests just to merge them. Auto-merge can be enabled by a user with permission to merge and on pull requests that have unsatisfied merge requirements. For more information, see "Automatically merging a pull request."

You can customize the types of notifications you want to receive from individual repositories. For more information, see "Configuring notifications."

GitHub Mobile filtering allows you to search for and find issues, pull requests, and discussions from your device. New metadata for issues and pull request list items allow you to filter by assignees, checks status, review states, and comment counts.

GitHub Mobile beta is available for GitHub Enterprise Server. Sign in with our Android and iOS apps to triage notifications and manage issues and pull requests on the go. Administrators can disable mobile support for their Enterprise using the management console or by running ghe-config app.mobile.enabled false. For more information, see "GitHub Mobile."

By precomputing checksums, the amount of time a repository is under the lock has reduced dramatically, allowing more write operations to succeed immediately and improving monorepo performance.

The latest release of the CodeQL CLI supports uploading analysis results to GitHub. This makes it easier to run code analysis for customers who wish to use CI/CD systems other than GitHub Actions. Previously, such users had to use the separate CodeQL runner, which will continue to be available. For more information, see "About CodeQL code scanning in your CI system."

GitHub Actions now supports skipping push and pull_request workflows by looking for some common keywords in your commit message.

Check annotations older than four months will be archived.

Following feedback, display of Code Scanning results on a pull request without submitting with a pull request ID will remain supported. For more information, see "Configuring code scanning" and "Configuring CodeQL code scanning in your CI system.

SARIF upload support increased to a maximum of 5000 results per upload.

You can specify multiple callback URLs while configuring a GitHub App. This can be used in services with multiple domains or subdomains. GitHub will always deny authorization if the callback URL from the request is not in the authorization callback URL list.

The GitHub App file permission has been updated to allow an app developer to specify up to 10 files for read-only or read-write access that their app can request access to.

CodeQL now supports more libraries and frameworks for a variety of languages (C++, JavaScript, Python,Java, Go). The CodeQL engine can now detect more sources of untrusted user data, which improves the quality and depth of the code scanning alerts. For more information, see "About CodeQL."

When configuring a GitHub App, the authorization callback URL is a required field. Now, we allow the developer to specify multiple callback URLs. This can be used in services with multiple domains or subdomains. GitHub will always deny authorization if the callback URL from the request is not in the authorization callback URL list.

Delete an entire directory of files, including subdirectories, from your web browser. For more information, see "Deleting a file or directory."

Include multiple words after the # in an issue, discussion, or pull request comment to further narrow your search.

When you’re writing an issue, pull request, or discussion comment the list syntax for bullets, numbers, and tasks autocompletes after you press return or enter.

The code scanning API allows users to upload data about static analysis security testing results, or export data about alerts. For more information, see the code scanning API reference.

The GitHub Apps API for managing installations has now graduated from an API preview to a generally available API. The preview header is no longer required to access these endpoints.

MEDIUM Under certain circumstances, users who were removed from a team or organization could retain write access to branches they had existing pull requests opened for.

Packages have been updated to the latest security versions.

All known issues from Release Candidate 1 have been fixed, except those listed in the Known Issues section below.

On the "Configure Actions and Packages" page of the initial installation process, clicking on the "Test domain settings" button did not complete the test.

Running ghe-btop failed with an error and cannot find a babeld container.

MySQL could reload and cause downtime if you change auto failover settings.

After upgrading, a mismatch of internal and external timeout values created service unavailability.

Expected replication delays in MSSQL generated warnings.

Link to "Configuring clustering" on the Management Console was incorrect.

When creating or editing a pre-receive hook, a race condition in the user interface meant that after selecting a repository, files within the repository were sometimes not populated in files dropdown.

When an IP address is added to a whitelist using "Create Whitelist Entry" button, it could still be shown as locked out.

References to the "Dependency graph" and "Dependabot alerts" features were not shown as disabled on some repositories.

Setting an announcement in the enterprise account settings could result in a 500 Internal Server Error.

HTTP POST requests to the /hooks endpoint could fail with a 401 response due to an incorrectly configured hookID.

The build-server process failed to clean up processes, leaving them in the defunct state.

spokesd created excessive log entries, including the phrase "fixing placement skipped".

While upgrading Actions the upgrade could fail if the instance could not make self-requests via its configured hostname.

Upgrading from 2.22.x to 3.1.0.rc1 could result in a database migration error relating to the BackfillIntegrationApplicationCallbackUrlsTransition data transition.

Access to a repository through the administrative shell using ghe-repo <owner>/<reponame> will hang. As a workaround, use ghe-repo <owner>/<reponame> -c "bash -i" until a fix is available in the next version.

The GitHub Packages npm registry no longer returns a time value in metadata responses. This was done to allow for substantial performance improvements. We continue to have all the data necessary to return a time value as part of the metadata response and will resume returning this value in the future once we have solved the existing performance issues.

On a freshly set up GitHub Enterprise Server without any users, an attacker could create the first admin user.

Custom firewall rules are not maintained during an upgrade.

Git LFS tracked files uploaded through the web interface are incorrectly added directly to the repository.

Issues cannot be closed if they contain a permalink to a blob in the same repository where the file path is longer than 255 characters.

When "Users can search GitHub.com" is enabled with GitHub Connect, issues in private and internal repositories are not included in GitHub.com search results.

Upgrading an instance that has previously ran a 2.13 release, but not a 2.14 release, results in a database migration error relating to the AddRepositoryIdToCheckRuns data transition.

After upgrading from 3.0.x to 3.1.x, in some cases GitHub Actions can fail with an error: An unexpected error occurred when executing this workflow. To workaround this problem, connect to the administrative shell (ssh) and run:

ghe-actions-console -s actions -c "Queue-ServiceJob -JobId 4DB1F4CF-19FD-40E0-A253-91288813DE8B"

When a replica node is offline in a high availability configuration, GitHub Enterprise Server may still route GitHub Pages requests to the offline node, reducing the availability of GitHub Pages for users.

Resource limits that are specific to processing pre-receive hooks may cause some pre-receive hooks to fail.

GitHub Enterprise Server 2.20 was discontinued on March 2, 2021. That means that no patch releases will be made, even for critical security issues, after this date. For better performance, improved security, and new features, upgrade to the newest version of GitHub Enterprise Server as soon as possible.

GitHub Enterprise Server 2.21 will be discontinued on June 9, 2021. That means that no patch releases will be made, even for critical security issues, after this date. For better performance, improved security, and new features, upgrade to the newest version of GitHub Enterprise Server as soon as possible.

Starting with GitHub Enterprise Server 2.21.0 two legacy GitHub Apps-related webhook events have been deprecated and will be removed in GitHub Enterprise Server 3.2.0. The deprecated events integration_installation and integration_installation_repositories have equivalent events which will be supported. More information is available in the deprecation announcement blog post.

Starting with GitHub Enterprise Server 2.21.0 the legacy GitHub Apps endpoint for creating installation access tokens was deprecated and will be removed in GitHub Enterprise Server 3.2.0. More information is available in the deprecation announcement blog post.

GitHub no longer supports the OAuth application endpoints that contain access_token as a path parameter. We have introduced new endpoints that allow you to securely manage tokens for OAuth Apps by moving access_token to the request body. While deprecated, the endpoints are still accessible in this version. We intend to remove these endpoints on GitHub Enterprise Server 3.4. For more information, see the deprecation announcement blog post.

GitHub Actions will remove support for referencing actions using the shortened version of a git commit SHA. This may cause some workflows in your repository to break. To fix these workflows, you will need to update the action reference to use the full commit SHA. For more information, see "Security hardening for GitHub Actions."

Beginning in GitHub Enterprise Server 3.1, we will begin discontinuing support for Xen Hypervisor. The complete deprecation is scheduled for GitHub Enterprise Server 3.3, following the standard one year deprecation window.