Wikimedia Platform Engineering
From MediaWiki.org
The Wikimedia Foundation Platform Engineering team is responsible for the MediaWiki core, managing work from volunteer developers, as well as for providing services that are used by multiple technology teams. The Platform Engineering group is managed by Rob Lanphier. It is part of the Wikimedia Foundation's engineering department.
See also a series of articles presenting the Platform engineering group:
- What is “Platform Engineering”?
- Data analytics at Wikimedia Foundation
- Technical Liaison; Developer Relations (TL;DR) (former name of the "Engineering community team")
- The MediaWiki Core group
- Meet the Analytics Team
- Introducing the Multimedia team
MediaWiki Core[edit | edit source]
Team
-
Tim Starling (lead)
Responsibilities[edit | edit source]
- Manage the MediaWiki release cycle
- Ensure that MediaWiki core is meeting the evolving needs of the website
- Make quality MediaWiki releases available for others outside of the Wikimedia Foundation
- Develop and document a clear set of APIs so that external developers can create applications that easily interface with MediaWiki
Current activities[edit | edit source]
Main projects[edit | edit source]
| Project | Description | Project lead & team | Status |
|---|---|---|---|
| MediaWiki 1.22 |
[edit]
Deployment of MediaWiki 1.22wmf versions and release of MediaWiki 1.22 |
[edit]
Sam Reed |
Status: (2013-09-monthly)
In September, MediaWiki 1.22wmf16 through 1.22wmf19 were deployed to the production Wikimedia Foundation cluster.
|
| Site performance and architecture |
[edit]
Various initiatives geared toward improving site responsiveness, decreasing resource consumption, and/or improving site maintainability |
[edit]
Tim Starling, Asher Feldman, Ori Livneh, Aaron Schulz, Antoine Musso, Andrew Garrett |
Status: (2013-08-20)
Ceph: This morning (Pacific time) we enabled multi-write to both Ceph and Swift (ie: Ceph will be as up to date as the main file store constantly). Later this week (probably Thursday) around the same time (early early Pacific morning) we'll switch the 'master' store to Ceph. ie: you shouldn't see any issues now on the user facing end, but you might later this week (but probably not, based on what we saw this morning).
|
| Admin tools development |
[edit]
Feature development and bugfixes on tools for stewards and local power-users |
[edit]
James Forrester (Product Manager), Chris Steipp, Peter Gehres, Tim Starling |
Status: (2013-09-monthly)
Although this activity is still officially on hold, several bug fixes were committed this month by community members. The GSoC project to implement a simple Bayesian Filter extension was successfully completed.
|
| Search |
[edit]
Search infrastructure maintenance and overhaul |
[edit]
Tim Starling, Chad Horohoe, Nik Everett |
Status: (2013-09-monthly)
In September, we expanded the new CirrusSearch back-end to a number of wikis. Italian Wiktionary, Catalan Wikipedia and English Wikisource are all running CirrusSearch now. Additionally, we deployed to all "closed" wikis. Further feature refinement and bugfixing are ongoing, with roughly 2 to 3 deployments a week.
|
| Auth systems |
[edit]
Improvement of authentication and authorization systems, and adding new auth mechanisms such as OpenID and OAuth |
[edit]
Chris Steipp, Tim Starling, Brad Jorsch, Aaron Schulz |
Status: (2013-09-monthly)
The team improved the user interface of OAuth and deployed these changes to mediawiki.org and test.wikipedia.org. We hope to test and refine the extension with third party developers, and subsequently deploy to all wikis. An initial review of Extension:OpenID was performed, and several issues were brought to the attention of the extension maintainer. Several bugs with CentralAuth/SUL were also fixed.
|
| Wikimania Scholarship app |
[edit]
Update and host the Wikimania Scholarship application. |
[edit]
Bryan Davis, Chad "^demon" Horohoe, Katie Filbert |
Status: (2013-10-30)
At midpoint of cleanup sprint approximately half the listed tasks are completed, but these tasks encompass only about one third of the total work. Progress is being made and the only roadblock is managing non-sprint related interruptions. The project should be in good shape for a progress review meeting in the later half of next week.
|
Continuous responsibilities[edit | edit source]
| Project | Description | Project lead & team | Status |
|---|---|---|---|
| Shell requests |
[edit]
Ongoing processing of Bugzilla requests that require shell access to fix |
[edit]
Sam Reed, |
Status: (2013-10-16)
Mblacklabad. Or
|
| Security auditing and response |
[edit]
Inspection of code and training of developers |
[edit]
Chris Steipp |
Status: (2013-09-monthly)
The team responded to reported issues, and released MediaWiki 1.21.2, 1.20.7 and 1.19.8 security releases to fix several issues in core and extensions.
|
Planned or backburner projects[edit | edit source]
| Project | Description | Project lead & team | Status | |
|---|---|---|---|---|
| API maintenance |
[edit]
Perennial maintenance of the MediaWiki API |
[edit]
Sam Reed |
Status: (2012-12-monthly)
,help
|
|
| Replicated session handling | Create a project page for this activity (how-to) | |||
| App-level monitoring |
[edit]
Implementation of application-level parameters monitoring |
[edit]
Rob Lanphier, Sam Reed |
Status: (2012-08-monthly)
support
|
|
| AcademicAccess |
[edit]
An authentication proxy to allow selected users to access third-party academic publishing sites |
[edit]
Chad Horohoe, Ryan Lane |
Status: (2013-05-03)
Academic Access to Poor People
|
|
| Configuration management |
[edit]
A core feature of MediaWiki to handle a wiki's configuration options through a web interface |
[edit]
Chad Horohoe |
Status: (2011-07-24)
This project was mainly on hold in July.
|
|
| Template profiling | Create a project page for this activity (how-to) | |||
| Continuous external storage recompression | Create a project page for this activity (how-to) | |||
| Database sharding |
[edit]
Enabling sharded tables in MediaWiki |
[edit]
Aaron Schulz, Asher Feldman |
Status: (2012-07-monthly)
Aaron Schulz wrote a new class (ExternalRDBStore) used for sharding tables in MediaWiki, and is now in bugfixing mode. He also wrote a patch to shard some of the tables associated with FlaggedRevs as a first use of this class. Asher Feldman is currently investigating hardware requirements for utilizing sharding.
|
|
| Code review management |
[edit]
Review of changes made to the MediaWiki core code and extensions used on Wikimedia wikis. |
[edit]
Rob Lanphier, Tim Starling, Chad Horohoe, Sam Reed, Aaron Schulz |
Status: (2012-09-monthly)
Volunteers have been of a lot of help in managing the wiki configuration (operations/mediawiki-config) repository. The WMF analytics code review graphs show an uptick in patchsets awaiting review at the end of September, and a Signpost analysis showed (among other statistics) that WMF staff provide 86% of first reviews for core patchsets, and just five staffers collectively account for about 55% of that total.
|
|
| REST proposal |
[edit]
REST implementation of MediaWiki's web API for greater flexibility, usability, and standardization |
[edit]
Federico Lucignano (Wikia) |
Status: (2013-02-monthly)
Yuri started gathering requirements for the RESTful (content-oriented) API as part of the overall planning for the API v2.0 roadmap. Also, Wikia's focused R&D sprint has led to remove all the remaining obstacles identified previously and Wikia has identified a new, larger dedicated Product team to get to a final implementation of the REST API following the directions set by the existing prototype and internal RFC; the team (API/Data) is the same one in charge of Search and all the related API's, this will ensure a better integration of this product into the new API strategy. At the moment of writing the knowledge transfer required for the team to start this new phase has just begun and the team has to first to complete the current work on another project already in progress. In the meantime, Wikia will make an RFC public for review and feedback.
|
|
| HipHop deployment |
[edit]
Support in MediaWiki for the HipHop source code transformer, and deployment to the Wikimedia cluster |
[edit]
Tim Starling, Sam Reed, Chad Horohoe, Mark Bergsma, Domas Mituzas |
Status: (2013-07-monthly)
HipHop work was mainly on hold in July, with the exception of some minor work on virtual machines.
|
|
Recently completed projects[edit | edit source]
| Project | Description | Project lead & team | Status |
|---|---|---|---|
| Disk-backed object cache |
[edit]
A supplement to memcache-based object cache, which will allow us to greatly increase the amount of space dedicated to caching parsed pages. |
[edit]
Rob Lanphier, Tim Starling |
Status: (2011-07-25)
To improve the MySQL-based version of this system, Domas Mituzas suggested to split the cache into several tables, which Tim Starling implemented in MediaWiki. The system was then deployed on July 11th and the cache has been filling up since then, thus increasing the parser cache hit ratio from about 30% to 80%. Possible future steps include adding previous page revisions to the cache.
|
| Heterogeneous deployment |
[edit]
A system allowing the deployment of different versions of MediaWiki to different production wikis |
[edit]
Rob Lanphier, Tim Starling, Aaron Schulz |
Status: (2012-04-17)
This feature is now deployed on Wikimedia's sites, and we use it to do staged rollouts of new versions of MediaWiki to our wikis.
|
| MediaWiki 1.18 |
[edit]
Deployment and release of MediaWiki 1.18 |
[edit]
Sam Reed, Tim Starling (pretty much everyone else is involved) |
Status: (2011-11-30)
Developers sprinted to fix the last blocker bugs, and the ones uncovered by testers. Sam Reed announced the first beta release and first release candidate of MediaWiki 1.18. Mark Hershberger followed up on comments on the English Wikipedia's Village pump related to the deployment of 1.18, bugs reported in bugzilla, and installation and upgrade reports. Sam announced the final 1.18.0 release on November 27th, as well as the 1.17.1 security release.
|
| MediaWiki 1.19 |
[edit]
Deployment and release of MediaWiki 1.19 |
[edit]
Rob Lanphier |
Status: (2012-05-03)
We have now released the tarball of MediaWiki 1.19. Work on 1.19 is now complete, barring future security updates, and engineers are now working on MediaWiki 1.20.
|
| MediaWiki 1.20 |
[edit]
Release of MediaWiki 1.20 |
[edit]
Sam Reed, Mark Hershberger |
Status: (2013-08-monthly)
iphone
|
| MediaWiki 1.21 |
[edit]
Deployment of MediaWiki 1.21wmf versions and release of MediaWiki 1.21 |
[edit]
Sam Reed |
Status: (2013-06-monthly)
No new releases were made for 1.21 in June of 2013.
|
| Wikivoyage migration |
[edit]
Migrating data and code to set up the new Wikimedia travel site |
[edit]
Erik Moeller, Daniel Zahn, Chris Steipp, Sam Reed, Matthias Mullie |
Status: (2013-01-monthly)
Wikivoyage officially launched on January 15. Most of the Wikimedia Foundation's involvement was completed in November, but some minor bugfixing was done in support of the official launch.
|
| VipsScaler |
[edit]
Deploy the VipsScaler extension to the Wikimedia cluster |
[edit]
Tim Starling, Bryan Tong Minh |
Status: (2013-08-monthly)
This month saw a significant change to the structure and organization of browser tests, with tests and builds for CirrusSearch, UniversalLanguageSelector, and VisualEditor following the example of MobileFrontend and now residing in the git repositories for those extensions, rather than in the /qa/browsertests repository. This creates opportunities for more frequent and more accurate Jenkins builds of the tests, while also reducing the overhead required for analyzing test failures.
|
| Lua scripting |
[edit]
Support for Lua as a scripting language for MediaWiki end-users |
[edit]
Tim Starling, Brad Jorsch (anomie) |
Status: (2013-04-monthly)
Some bugs were fixed and internationalization changes merged this month; no major changes were made. The community continues to develop Lua-based templates, such as the citation templates on the English Wikipedia.
|
| Git conversion |
[edit]
Conversion of the MediaWiki source repository from Subversion to Git |
[edit]
Chad Horohoe |
Status: (2013-07-monthly)
With the migration of pywikipediabot from Subversion to Git, we were able to switch svn.wikimedia.org to read-only mode, thus completing this migration. We plan to keep the Subversion service around indefinitely for archival purposes, and can still migrate any dormant project that hasn't already been migrated on request.
|
| Wikidata deployment |
[edit]
Deployment of Wikidata related core modifications |
[edit]
Daniel Kinzler, Tim Starling, Sam Reed, Chad Horohoe |
Status: (2013-05-monthly)
Wikidata continued to roll out updates to the client (what lives on the various project wikis) and repository (what powers wikidata.org) software; see the wikidata newsletter for more information: May 3rd and May 24th. Additionally, there was Operations-related work to create helpful URL redirects for entity matching along with fixes to the logging of autopatrolled edits (which was causing massive database usage and issues when doing archival dumps). These URL redirects and autopatrol fixes should go out in June.
|
Quality Assurance[edit | edit source]
Responsibilities[edit | edit source]
- Ensure that all new code that is released meets quality standards
- Triage, manage and resolve all bug reports
Team[edit | edit source]
-
Chris McMahon
Working closely with Michelle Grover in Mobile
Current activities[edit | edit source]
| Project | Description | Project lead & team | Status |
|---|---|---|---|
| Quality Assurance |
[edit]
Software testing and quality assurance for Wikimedia engineering activities |
[edit]
Chris McMahon, TBD |
Status: (2013-09-monthly)
This month, we wrapped up Rachel Thomas' Outreach Program for Women internship successfully. Rachel helped us extend our browser test coverage of VisualEditor. Besides our ongoing collaboration with Wikimedia Foundation development projects, we are also engaging the greater community on the QA mailing list, where we discuss both code contributions and general QA topics.
|
| Beta cluster |
[edit]
Establish a functional, production-like environment in Wikimedia Labs suitable for final-stage testing of new features. |
[edit]
Chris McMahon, Antoine Musso, Sam Reed, Faidon Liambotis |
Status: (2013-09-18)
The Flow MediaWiki extension has been enabled on the beta cluster. Will let the QA team the possibility to write browser test in a context close to production. bug 53061
|
| Continuous integration |
[edit]
A testing platform to provide continuous quality control for MediaWiki |
[edit]
Antoine Musso, Chad Horohoe, Timo Tijhof |
Status: (2013-09-19)
For the last few weeks, Antoine has been working toward having jobs running on a second machine. Today, all jobs are ready to freely roam on the Jenkins slave we have and Antoine started migrating the pep8 and pyflakes jobs.
|
| Browser testing |
[edit]
Automated testing of MediaWiki within various browsers to improve quality |
[edit]
Chris McMahon, Željko Filipin, Michelle Grover |
Status: (2013-09-monthly)
This month saw significant improvements to both coverage and speed in our tests for VisualEditor. We are collaborating with the Language team on browser tests for the UniversalLanguageSelector extension and Translatewiki.net. We created our first tests for the new Flow feature and are in the process of supporting Flow fully in a reference test environment. We presented yet another of our ongoing series of training sessions, this one live in San Francisco.
|
Engineering Community Team[edit | edit source]
Formerly "TL;DR" (Technical Liaisons & Developer Relations) Old archived notes from weekly meetings (caution: messy)
Team[edit | edit source]
-
Quim Gil (interim lead)
-
Sumana Harihareswara (lead — on leave)
Responsibilities[edit | edit source]
- Attract and support volunteer technologists so that they can become an even more powerful force in driving improvement on the website
- Help nurture volunteer-developed extensions and widgets
- Provide communications support for all groups in the engineering department
- Develop clear documentation for MediaWiki so that new staff, volunteer developers and external users have a smooth ramp-up process to becoming MediaWiki developers
Current activities[edit | edit source]
See also Engineering Community Team meetings notes.
| Project | Description | Project lead & team | Status |
|---|---|---|---|
| Bug management |
[edit]
Management of Wikimedia's bug tracker. |
[edit]
Andre Klapper |
Status: (2013-10-monthly)
Bugzilla now offers a new guided bug entry form which will make creating good bug reports easier for newcomers. Bugzilla now also displays metadata changes of a bug report inline for all logged-in users so it can be seen (without clicking on "History") in the comments who changed a value of a field. Daniel Zahn upgraded Wikimedia Bugzilla to latest version 4.2.7. Legoktm mass-imported about 400 Pywikibot tickets from Sourceforge to Bugzilla. On a related note, Amir ran a PyWikibot Bug Triage resulting in nearly 100 tickets receiving updates. Furthermore, Andre investigated Wikimedia Bugzilla's customizations in CSS and code in order to clean up and sync with the upstream code base to simplify current maintenance and also make a potential future upgrade of Wikimedia Bugzilla from version 4.2 to 4.4 easier.
|
| Mentorship programs |
[edit]
Google Summer of Code, LevelUp, Outreach Program for Women, and other programs that mentor contributors |
[edit]
Quim Gil, Sumana Harihareswara |
Status: (2013-09-monthly)
18 out of the 20 Google Summer of Code projects have passed the program evaluation, as well as the one Outreach Program for Women project (read our announcement and blog post). These numbers are unprecedented and we have to ensure that they are not just occasional results but a trend.
Wrap-up reports from the projects:
|
| Technical communications |
[edit]
An activity aiming to incrementally improve communications between the communities of developers and users |
[edit]
Guillaume Paumier |
Status: (2013-09-monthly)
Guillaume Paumier wrapped up work on supporting the deployment of VisualEditor, and resumed regular activities like preparing the Tech newsletter and ongoing communications support for the engineering staff.
|
| Volunteer coordination and outreach |
[edit]
Coordination of volunteer efforts, technical outreach and recruitment of new volunteers |
[edit]
Sumana Harihareswara, Quim Gil, Guillaume Paumier |
Status: (2013-09-monthly)
Together with XWiki and Tiki, we submitted a Wiki devroom proposal for FOSDEM, the biggest open source source conference in Europe. We are also preparing a proposal for a stand, lead by volunteers at the nascent Wikimedia Belgium chapter. The overall goal is to achieve a good MediaWiki & Wikimedia tech gathering in Brussels next February. We are also supporting the organization of the MediaWiki Architecture Summit in San Francisco on 23-24 January, 2014.
|
Planned or backburner projects[edit | edit source]
Recently completed projects[edit | edit source]
| Project | Description | Project lead & team | Status |
|---|---|---|---|
| MediaWiki architecture document |
[edit]
An overview of MediaWiki's architecture and decisions that led to it |
[edit]
Guillaume Paumier, Sumana Harihareswara |
Status: (2012-04-17)
The community awaits the publication of Volume 2 of Architecture of Open Source Applications, probably in a matter of days. It will include our now-finished chapter on MediaWiki's architecture.
|
| Wikimedia blog maintenance |
[edit]
Maintenance and new features for the Wikimedia blog |
[edit]
Guillaume Paumier, |
Status: (2012-04-monthly)
Guillaume Paumier made minor changes to the theme, like not displaying trackbacks and pings in the comments list. He investigated plugins to support multiple authors, photo galleries and multilingual posts, and is testing Piwik as a possible solution to record metrics. Chad Horohoe created git repositories for the WP-Victor theme and the WMBlog plugin in Gerrit, and imported their history from GitHub.
|
| Summer of Code 2011 |
[edit]
A sponsored community program allowing students to join the community as developers. |
[edit]
Sumana Harihareswara |
Status: (2012-11-20)
Neil Harris is now working on getting Kevin Brown's ArchiveLinks project completed and deployed. This activity page ("Summer of Code 2011") is now being retired; in the future, status updates will be on the Mentorship programs page.
|
| Summer of Code 2012 |
[edit]
A sponsored community program allowing students to join the community as developers. |
[edit]
Sumana Harihareswara, Gregory Varnum as backup organizational administrator |
Status: (2012-11-20)
The Wikimedia community is following up its Summer of Code participation with participation in the GNOME Outreach Program for Women, which offers paid internships for work in open source software projects. This follows the community's decision not to participate in Google Code-In due to a lack of mentors. Also, Sumana Harihareswara posted a retrospective on our participation in Summer of Code 2012, concluding that we're improving at student retention but need to get better at producing something useful at the end of a mentorship period, and offering suggestions on how to improve our processes in the future. This activity page ("Summer of Code 2012") is now being retired; in the future, status updates will be on the Mentorship programs page.
|
| Wikimedia Foundation engineering 20% policy |
[edit]
A policy and practice whereby WMF engineering staff spent at least 20% of their work time on tasks that directly serve the Wikimedia developer and user community |
[edit]
Sumana Harihareswara, Rob Lanphier, All engineering staff |
Status: (2012-11-monthly)
Sumana Harihareswara has developed Mentorship programs/LevelUp, a new program in which WMF engineers and systems administrators will grow their and the community's capacity for the long term. This replaces 20% time, which is now defunct.
|
Multimedia[edit | edit source]
Team[edit | edit source]
Responsibilities[edit | edit source]
- Improve MediaWiki's and Wikimedia's support for audio, video, image, and other media files
Current activities[edit | edit source]
Main projects[edit | edit source]
| Project | Description | Project lead & team | Status |
|---|---|---|---|
| Multimedia |
[edit]
Handling of images, sound and video files, from infrastructure to contributions, curation, publishing and viewing. |
[edit]
Rob Lanphier (engineering director)Fabrice Florin (product manager), Bryan Davis, Mark Holmquist, Faidon Liambotis, Aaron Schulz, Gergő Tisza |
Status: (2013-09-monthly)
In September, we continued to expand our multimedia team and updated our multimedia plan for the coming year (see slides). Mark Holmquist continued development on the Media Viewer to improve the image viewing experience, based on designs by May Tee-Galloway and Jared Zimmerman. We also made good progress on the Beta Features project, which will invite users to test, give feedback, and use a range of new features in real-world settings. We aim to have first beta versions of both products ready by the end of the October. New employee Bryan Davis started work on several multimedia platform bugs, and Summer contractor Jan Gerber completed his work on the TimedMediaHandler extension. To discuss these features and keep up with our work, we invite you to join the new multimedia mailing list. Last but not least, we are also recruiting for a senior software engineer position on our team.
|
