Extension:GnuplotBasic

MediaWiki extensions manual

GnuplotBasic Release status: unmaintained
Implementation	Tag
Description	Define Basic Gnuplot script (plus optional plotdata) and render graph image
Author(s)	Gérard de Smaeletalk
Latest version	1.0.0 (2009-11-18)
MediaWiki	1.13, 1.14, 1.15...
License	none
Download	No link
Translate the GnuplotBasic extension if it is available at translatewiki.net
Check usage and version matrix; code metrics

Purpose

This GnuplotBasic extension allows you to produce and display graph images (like mathematical functions, line charts, histograms etc.) within a MediaWiki instance. It uses the (free) plotting program Gnuplot for creating these graphs. It allows for 2- or 3-dimensional graphs, all within the potentials of the Gnuplot program itself. The graphs can be based on mathematical function(s) and they can use plotdata from the wiki page. GnuplotBasic works, together with the Gnuplot progam, fully within your MediaWiki Server environment.

Important: Before using this extension read the its future paragraph !!

Use

Wiki markup structure for a GnuplotBasic call

Include the GnuplotBasic call between the tags <gnuplot> ... </gnuplot>.

A GnuplotBasic call in wiki markup consists of a plotscript-part plus an optional plotdata-part (see structure on the right). The plotscript has the Gnuplot commands, you can find these in the Gnuplot manual. The plotdata has your data (in rows and columns) to be plotted. The plotscript-part always comes before the plotdata-part.

If you use plotdata include it within <gnuplot> ... </gnuplot> and within <plotdata> ... </plotdata>. There can be no more Gnuplot commands after </plotdata>.

The only required Gnuplot command in the plotscript-part is the 'plot' command, e.g. plot sin(x). The 'plot' command should be the last plotscript command in order for any settings to take effect.

Sample plots simple

Here are some simple samples of what you can do with Gnuplot with the use of GnuplotBasic:

Wiki Markup	Display
You can plot functions without data e.g. <gnuplot> set terminal png small transparent set size 0.4, 0.4 set title "GnuplotBasic Sample 1" plot sin(x) </gnuplot>
You can plot graphs with inline plotdata e.g. <gnuplot> set terminal png small transparent set title "GnuplotBasic Sample 2" plot '-' using 1:2 t 'line 1' with linesp lt 1 lw 1, \ '-' using 1:3 t 'line 2' with linesp lt 2 lw 1 <plotdata> #num line1 line2 1 2 3 2 4 9 3 8 27 </plotdata> </gnuplot>

• You can put comment in your GnuplotBasic call after the "#" character. Best practice is to do this at the beginning of a line.
  Comment can be used both in the plotscript-part and the plotdata-part. This can be especially helpful for documenting columnheaders when using plotdata.

Sample plots more advanced

Here are some more advanced samples of what you can do with Gnuplot with the use of GnuplotBasic:

Wiki Markup	Display
You can plot graphs with inline plotdata e.g. <gnuplot> set terminal png transparent font verdana 8 set title "GnuplotBasic Sample 3" set xlabel "Month of 2008" set ylabel "Amount (Euro)" set auto x plot '-' using 1:2 title 'planned',\ '-' using 1:3 title 'interpolated',\ '-' using 1:4 title 'spent' <plotdata> # month planned interp spent 1 300000 42775 50000 2 300000 85550 83200 3 300000 128325 137650 4 300000 171100 187655 5 300000 213875 246585 6 300000 256650 265475 7 300000 299425 309870 8 300000 342200 363976 9 300000 384975 403692 10 300000 427750 459872 11 300000 470525 491234 12 300000 513300 536829 </plotdata> </gnuplot>
You can plot graph histograms with inline plotdata e.g. <gnuplot> set terminal png transparent font verdana 8 set title "GnuplotBasic Sample 4" set auto x set grid set yrange [0:350000] set style data histogram set style histogram cluster gap 2 set style fill solid border -2 set boxwidth 0.9 set xtic rotate by -75 plot '-' using 2:xtic(1) ti col, \ '' using 3 ti col, \ '' using 4 ti col, \ '' using 5 ti col <plotdata> # Immigration statistics Region Denmark Netherlands Norway Sweden 1891-1900 30770 205290 8731 21179 1901-1910 73379 190505 249534 7 1911-1920 61897 66395 95074 4813 1921-1930 49610 68531 97249 227734 1931-1940 12623 4740 3960 17026 1941-1950 38809 10100 10665 7571 1951-1960 51121 22935 21697 9985 1961-1970 45237 15484 17116 23539 </plotdata> </gnuplot>

Note that these two samples use font verdana 8. Advanced (True-Type) fonts are required when rotating text (you'll probably need rotating text making histograms). Make sure the Gnuplot program can find the fonts you use in the plotscript, see installation.

Installation and configuration

Got you interested with these samples? Here's how to install it.
Installation consists of a few steps. Note that in the descriptions below $IP is the root install path of your MediaWiki.

• Install the Gnuplot plotting program

You'll need a version of Gnuplot. You can download it from the Gnuplot website. Follow its installation instructions suiting your choice of platform (Windows or Unix). Check the environment variable GDFONTPATH if you need or like to use more advanced fonts rather than the build in ones: you have to install the fonts on your system and add

putenv("GDFONTPATH=[...]"); // where [...] is the font path

or similar into your $IP/LocalSettings.php.

• Install this extension

Save the PHP script below as file GnuplotBasic.php in the directory $IP/extensions/GnuplotBasic/

• Add the following lines to the file LocalSettings.php

require_once("$IP/extensions/GnuplotBasic/GnuplotBasic.php");

$wgGnuplotCommand = "<yourGnuplotPath>";

• Define <yourGnuplotPath> as absolute path to the executable of the Gnuplot program e.g.:

$wgGnuplotCommand = "C:\\wamp\\bin\\gnuplot\\bin\\pgnuplot.exe";   //for windows

$wgGnuplotCommand = "/usr/bin/gnuplot";                            //for unix

• Make sure that directory $IP/images is writable.

Configuration options
Optionally you can configure three other settings for GnuplotBasic in LocalSetting.php. If you're comfortable with the default settings you can omit these settings.

• $wgGnuplotDefaultTerminal, the default GnuplotBasic terminal setting is "set terminal png transparent small";.
  You can change the default terminal used for all graphs e.g.:

$wgGnuplotDefaultTerminal = "set terminal png transparent font verdana 8";

Note: This default setting can be overruled in a plotscript.

• $wgGnuplotDefaultSize, the default GnuplotBasic size setting is "set size 0.5, 0.5".
  You can change the default size used for all graphs e.g.:

$wgGnuplotDefaultSize = "set size 0.4, 0.4";

Note: This default setting can be overruled in a plotscript.

• $wgGnuplotRefreshRate, the default GnuplotBasic refresh rate setting is "-1".
  GnuplotBasic creates graph images in directory images/gnuplot with unique (automatically calculated) filenames based on the content of the <gnuplot> call (including its optional plotdata). You can set the behavior of GnuplotBasic in case the graph image file already exists. You can define whether or not GunuplotBasic re-creates a new graph image when it already exists by defining the refresh rate. Some example settings:

$wgGnuplotRefreshRate = "-1";         //Refresh graph never when it hasn't changed (default)
$wgGnuplotRefreshRate = "24*60*60";   //Refresh graph if it is older than 24*60*60 seconds (one day)
$wgGnuplotRefreshRate = "0";          //Refresh graph always, each time the page is viewed

Note: The default refresh rate = "-1" because this prevents unnecessary re-creation of graph images each time a page is viewed and the content of the <gnuplot> call hasn't changed.

PHP Script

Cut and paste this whole PHP script and save it as GnuplotBasic.php in $IP/extensions/GnuplotBasic/.

<?php
/*****************************************************************************
* PURPOSE
*   Extension script to use Gnuplot in a MediaWiki instance.
*   Insert the Gnuplot code between the tags <gnuplot>...</gnuplot>.
*
*   Within <gnuplot> ... </gnuplot> this extension accepts (inline) plotdata 
*   within <plotdata> ... </plotdata>. If there is plotdata the source
*   is spilt in a plotscript-part and a plotdata-part. Note that <plotdata> is
*   not a MediaWiki hook.
*
* AUTHOR
*   Gérard de Smaele
*
* NOTES
*   Include this in the LocalSettings.php:
*   1) add 'require_once("extensions/GnuplotBasic/GnuplotBasic.php")'
*   2) Set $wgGnuplotCommand to the Gnuplot executable.
*      For windows: "C:\\Program files\\Gnuplot\\pgnuplot.exe"
*      Default:     "/usr/bin/gnuplot"
*   3) Set $wgGnuplotRefreshRate to desired refresh rate 
*      (in seconds) of a graph in case it already exists.
*      - (24*60*60) Refresh after one day (x seconds)
*      - (0)        Refresh always (zero)
*      - (-1)       Refresh never  (minus one)
*   4) Set $wgGnuplotDefaultTerminal to standard terminal e.g.
*      - set terminal png transparent small (default)
*      - set terminal png transparent font verdana 8
*******************************************************************************/
 
// Extension credits that will show up on Special:Version
$wgExtensionCredits['parserhook'][] = array(
	'name' => 'GnuplotBasic',
	'author' => 'Gérard de Smaele',
	'version' => '1.0.0',
	'description' => 'Adds the tag <code>&lt;gnuplot&gt;</code> to use Gnuplot',
	'url' => 'http://www.mediawiki.org/wiki/Extension:GnuplotBasic',
);
 
$wgGnuplotCommand = '/usr/bin/gnuplot'; // Unix
#$wgGnuplotCommand = 'C:\\Wamp\\bin\\gnuplot\\bin\\pgnuplot.exe';  // Windows
 
$wgGnuplotDefaultTerminal = 'set terminal png transparent small';
$wgGnuplotDefaultSize     = 'set size 0.5, 0.5';
$wgGnuplotRefreshRate     = -1; // Refresh never if gnuplot call hasn't changed (default)
 
// Avoid unstubbing $wgParser too early on modern (1.12+) MW versions, as per r35980
if ( defined( 'MW_SUPPORTS_PARSERFIRSTCALLINIT' ) ) {
	$wgHooks['ParserFirstCallInit'][] = 'wfGnuplotBasicExtension';
} else {
	$wgExtensionFunctions[] = 'wfGnuplotBasicExtension';
}
 
function wfGnuplotBasicExtension() {
	global $wgParser;
	$wgParser->setHook( 'gnuplot', 'renderPlot' );
	return true;
}
 
function renderPlot( $gnuplotsrc ) {
	global $wgGnuplotDefaultTerminal, $wgGnuplotDefaultSize;
	global $wgGnuplotCommand, $wgGnuplotRefreshRate;
	global $wgUploadDirectory, $wgUploadPath;
 
	// create directory for storing the plot files
	$gnuplotDir = "/gnuplot/";
	$dest = $wgUploadDirectory . $gnuplotDir;
	if( !is_dir( $dest ) ) {
		mkdir($dest, 0777);
		chmod($dest, 0777);
	}
	// make sure the path is correct for both Windows and UNIX
	chdir($dest);
	$dest = getcwd(); 
 
	// get the filename of the graph to be produced
	$name = getFileName( $gnuplotsrc );
	$graphname = $dest . DIRECTORY_SEPARATOR .$name;
	$fname = $graphname . ".tmp";  //gnuplot plotscript
	$dname = $graphname . ".dat";  //gnuplot plotdata
 
	$makeGraph = true;
	if( ( file_exists( $graphname ) ) and ( ( $wgGnuplotRefreshRate ) <> 0 ) ) {
		// the file already exists it won't be refreshed when
		// it is newer than the refreshrate OR setting set to never refresh
		if( (time() - filemtime($graphname)) < ($wgGnuplotRefreshRate) ) $makeGraph = false;
		if( ($wgGnuplotRefreshRate) == -1 ) $makeGraph = false;
	}
 
	if( $makeGraph ) {
		// Step 1: Check if there is plotdata defined
		$pos1 = strpos($gnuplotsrc, "<plotdata>");
		if( $pos1 !== false ){
			// split gnuplotsrc in a plotscript-part and a plotdata-part
			$pos2 = strpos($gnuplotsrc, "</plotdata>");
			if( $pos2 === false ) $pos2 = strlen($gnuplotsrc);
			$plotdata = substr($gnuplotsrc, $pos1+10, $pos2-($pos1+10));
			$gnuplotsrc = substr($gnuplotsrc, 0, $pos1);
			// replace gnuplot inline filename(s) '-' to the plotdata filename
			$gnuplotsrc = str_replace("'-'", "'$dname'", $gnuplotsrc);
			// save the plotdata-part 
			$handle2 = fopen($dname, 'w');
			fwrite($handle2, trim($plotdata));
			fclose($handle2); 
		}
 
		// Step 2: Parse the plotscript-part and do some basic syntax checking
		$unparsedScriptArray = explode( "\n", $gnuplotsrc );
		$parsedScriptArray = array_filter( $unparsedScriptArray, "parsePlotScript" );
		$gnuplotsrc = implode( "\n", $parsedScriptArray );
 
		// write the default settings and the input code from wiki into a
		// temporary file to be executed by gnuplot, then execute the command
		$handle = fopen($fname, 'w');
 
		// if terminal and size are not set in the gnuplot source we do it here
		if( strpos($gnuplotsrc, 'set terminal ') === false ) {
			fwrite($handle, $wgGnuplotDefaultTerminal . "\n");
		}
		if( strpos($gnuplotsrc, 'set size ') === false ) {
			fwrite($handle, $wgGnuplotDefaultSize . "\n");
		}
 
		fwrite($handle, "\nset output '" . $graphname . "'\n");
 
		// save the plotscript-part
		fwrite($handle, $gnuplotsrc . "\n");
		fwrite($handle, "exit" . "\n");
		fclose($handle); 
 
		// execute the gnuplot command, force use environment variables
		$cmdlinePlot = $wgGnuplotCommand . ' ' . $fname;
		$gdpath = getenv("GDFONTPATH");
		if( $gdpath !== false ) putenv("GDFONTPATH=$gdpath");
		$gdpath = getenv("GNUPLOT_FONTPATH");
		if( $gdpath !== false ) putenv("GNUPLOT_FONTPATH=$gdpath");
		shell_exec($cmdlinePlot);
 
		// cleanup temporary files
		unlink($fname);
		if( $pos1 !== false ) unlink($dname);
	}
 
	return '<p><b><img src="' . htmlspecialchars($wgUploadPath . $gnuplotDir . $name) .
		'" alt="GnuplotBasic Plot"></b></p>';
}
 
/***
 * Function: parsePlotScript
 * Purpose : Basic syntax checks and replaces unwanted and potentially harmful commands.
 *           Note that gnuplot allows far more advanced syntax than is allowed
 *           and recognised here. This should however suffice most graph needs from 
 *           within a MediaWiki instance. 
 * Input   : &$scriptline - a line from the scriptpart (by reference because we do 
 *           some modifications here).
 * Output  : false if the line ends up empty.
 */
function parsePlotScript( &$scriptline ) {
 
	$unwantedCommands = array(
		"`" => "",
		"set output " => "xxx xxxxxx ",
		"shell" => "xxxxx", // potentially harmful
		"system" => "xxxxxx", // potentially harmful
		"load " => "xxxx ", 
		"call " => "xxxx ", 
		"save " => "xxxx ", 
		"cd '" => "xx ", 
		"cd \"" => "xx ", 
		"update " => "xxxxxx "
	);
 
	$scriptline = trim($scriptline);
	if( strlen($scriptline) == 0 ) return false;
 
	// Step 1: Delete everything after the first comment character
	$p = strpos($scriptline, "#");
	if( $p !== false ) {
		$scriptline = substr($scriptline, 0, $p);
		$scriptline = trim($scriptline);
		if( strlen($scriptline) == 0 ) return false;
	}
	// Step 2: Replace unwanted commands with substitutes
	$scriptline = strtr($scriptline, $unwantedCommands);
	$scriptline = trim($scriptline);
	if( strlen($scriptline) == 0 ) return false;
 
	return true;
}
 
/***
 * Function: getFileName
 * Purpose : Determines the unique name of the output file.  
 *           The filename is computed by a hash function and therefore always unique 
 *           for the script plus its (optional) inline plotdata.
 *           The file extension is extracted from de format in the "set terminal" 
 *           command. If this format is not found or there is a syntax error it is 
 *           set to "xxx".
 * Input   : $gnuplotsrc - the <gnuplot> input code from the wiki markup
 * Output  : The file name with its extension
 */
function getFileName ( $gnuplotsrc ) {
	// determine the file format of the plotted graph image - default is png
	$format = "png";
	$tpos = strpos($gnuplotsrc, "set terminal ");
	if( $tpos !== false ) {
		$format = '';
		$tpos = $tpos + strlen("set terminal ");
		// extract the fileformat from this command, this is considered
		// to be the next word either ending with " ", "\n" or "\t"
		$done = false;
		do {
			$char = substr($gnuplotsrc, $tpos, 1);
			$tpos = $tpos + 1;
			if( $char !== false ) {
				if( ($char == " ") || ($char == "\n") || ($char == "\t") ) {
					$done = true;
				} else {
					$format .= $char;
				}
			} else {
				$done = true;
			}
		} while( !$done );
 
		$format = trim($format);
		if( (strlen($format) == 0) || (strlen($format) > 6) ) $format = "xxx";
	}
	$filename = md5($gnuplotsrc) . "." . $format;
	return $filename;
}

How does GnuplotBasic work?

Underneath you'll find a summary of the global workings of GnuplotBasic:

• A GnuplotBasic call is initiated from a wiki page written in wiki markup. A call starts with the <gnuplot> tag. GnuplotBasic receives a call every time a wiki page with this <gnuplot> tag is viewed by a user
• GnuplotBasic receives the full text from the call between <gnuplot> ... </gnuplot>
• From this full text the unique graph image filename is determined (e.g. 4011db...0072995.png)
• If the graph image already exists GnuplotBasic (re-)creates it (or not) depending on the defined refresh rate. If it doesn't exist GnuplotBasic creates it.
• If plotdata is defined between <plotdata> ... </plotdata> then
  GnuplotBasic saves this plotdata in a temporary file (e.g. 4011db...0072995.png.dat)
• GnuplotBasic does some basic checking of the plotscript-part and replaces unwanted commands with harmless/useless ones
• GnuplotBasic then saves the plotscript-part in another temporary file (e.g. 4011db...0072995.png.tmp)
• The command to execute the plotscript is then given to the Gnuplot program. The Gnuplot program creates the graph image (if the plotscript's syntax is correct).
• The temporary file(s) are deleted
• The graph image (e.g. 4011db...0072995.png) is then shown on the page.
• If anything has gone wrong, either a mistake in the plotscript syntax or an error occurred on the server, the text GnuplotBasic is shown.

Maintenance of your site

As described in the installation, GnuplotBasic creates graph images in the directory images/gnuplot/*.*. The filenames of these images are automatically determined from the full content of the <gnuplot> call (including its optional plotdata). This means that every time the content of a <gnuplot> call changes the filename of the created image also changes.

A filename is for example: 4011db4213c05eed3f30f6a350072995.png

The consequence is that over time the directory images/gnuplot becomes populated with unused (orphan) graph images. Periodically the wiki site administrator should do some cleanup in this directory. It is recommended to delete all the files in directory images/gnuplot/*.* once in a while. Graph images will be (re)created whenever a page is viewed and GnuplotBasic discovers the graph image doesn't exist.

About GnuplotBasic...

Its origin

This extension is inspired by and on two other gnuplot extensions: Gnuplot-1 and Gnuplot-2. These two have the same name but are different. Thanks to the author(s) of those extensions to get me going. I started to tweak and change one of them to suit my needs and eventually created this new extension with its own functions. I hope this one will be of use to others as well.

Its development

This extension is developed and tested with Gnuplot 4.2, MediaWiki 1.13 and 1.14 and PHP 5.2 on a Windows machine. It should work just as fine on other versions/platforms, as long as PHP 5.x is used.

Its design choices

The design choices I have made for GnuplotBasic.

• No files, inline plotdata only. GnuplotBasic only accepts plotdata that is part of the wiki markup. Handling external datafiles from an extension is error prone and the uploaded datafiles are hard to manage.
• No "set output" in plotscript-part to a user defined image filename. Unique (generated) image filenames based on the full content of the GnuplotBasic call only.
• RefreshRate option. This gives control over the rate of re-creating graph images. Having a unique image filename based on the content of the GnuplotBasic call enables this option.
• GnuplotBasic regards all text on each line in a GnuplotBasic call after the first "#" it finds as comment. This approach is a bit simpler than what the Gnuplot program actually offers but it is sufficient for what is needed using Gnuplot from within MediaWiki.
• GnuplotBasic does some basic parsing/filtering of the plotscript-part. Unwanted Gnuplot commands (and potentially dangerous ones that I've identified) are replaced by "x"-es. This can either cause a script not to work or display "x"-es in a label if that text happens to be the same as an unwanted Gnuplot command.
• GnuplotBasic is called from MediaWiki with the <gnuplot> tag because this is shorter typing, rather than for example <gnuplotbasic>.

Why Gnuplot in the first place? Well there is an alternative for producing simple graph images, and that is extension Google Chart for MediaWiki (gchart4mw). Its simplicity and good looking graphs make it an attractive option (don't miss the bug fix on the extensions talk-page if you're thinking of using it). It also supports pie-charts and Gnuplot doesn't. However the downside of this alternative is that it always requires an Internet connection from a user browser. Plus, more serious, all the plotdata is send over the Internet to Google to make a graph each time a page is viewed. This is not so nice if you work with confidential or privacy sensitive data. Therefore I chose Gnuplot. It works fully within the wiki server environment. But... using Gnuplot also comes with some challenges and limitations, please read security limitations. This is a trade-off, it's your decision to make.

Its security limitations

Using Gnuplot, with the help of GnuplotBasic, on a MediaWiki based wiki site (either a local or public site) has some risks because of Gnuplot's extended command vocabulary/syntax and its acceptance of abbreviated commands. Authors using this extension can willingly or unwillingly use Gnuplot commands that have the potential of harming your system (even though some of these commands are filtered out). I am using it on a intranet based wiki, my assumption is that my users have good intentions and are not seeking to harm the system. Therefore I am accepting this (very small) risk.

You can find the list of Gnuplot commands that GnuplotBasic filters out in the PHP script. They are found in function parsePlotScript in the definition of $unwantedCommands.

If you know of Gnuplot commands (or abbreviations) that I've missed that should be filtered out, please make note of that on the talk page of this extension. This way you can help make GnuplotBasic better!

Its name

It is called GnuplotBasic because:

• It does some basic syntax parsing/filtering of Gnuplot commands in the plotscript-part
• This parsing/filtering reduces what you can do with Gnuplot a little bit to its basic functions. If anyone wants to use Gnuplot's full capabilities this has to be done with Gnuplot's standard GUI interface.
• The name needed to be different from the two already existing gnuplot extensions to prevent confusion.

Its future

gnuplotBasic was released beginning of 2009. Almost at the same time the free pChart library was discovered. After that discovery a colleague and I were inspired to create a new extension, using this PHP chart library, to have better looking charts and to solve some of the issues with gnuplotbasic.

On 2009-11-11 this new chart extension pChart4mw (pChart for Mediawiki) was released. It serves as a good alternative for gnuplotBasic. As you can read on gnuplotbasic's discussion page there are several pros and cons in using gnuplotbasic just because is uses the external program gnuplot.

If you're interested in line, bar, pie, scatter, radar and bubble charts for Mediawiki have a look at what pChart4mw can do for you.

As of 2009-11-18 gnuplotBasic will no longer be maintained by its original author. It will remain available and posted as a Mediawiki extension "as-is". Anyone who feels inclined to further develop or maintain gnuplotBasic is welcome to do so and take ownership of this extension.

Gérard 06:30, 18 November 2009 (UTC)

References

The Gnuplot homepage where you will find Gnuplot's download, documentation, manuals, quick reference guide, samples, etc...

• http://www.gnuplot.info/
  The plotting options that Gnuplot offers are almost endless. It does take some time though to learn how to use it well (but the reward is worth the investment). Make sure authors on your wiki familiarize themselves properly with the workings of Gnuplot. The Gnuplot Graphical User Interface is a helpful tool for practicing and learning. If you are using Gnuplot on Unix you may need to build the Gnuplot program from its sources. Make sure you build it with the GD library (Graphics Display Library) for the necessary png, gif and/or jpeg support.

Helpful links on using Gnuplot

• http://t16web.lanl.gov/Kawano/gnuplot/index-e.html Not so frequently asked questions

Alternative chart extension:

• pChart4mw For line, bar, pie, scatter, radar and bubble charts for Mediawiki