About History Screenshots Demo Requirements Download Install MailList Acknowledgments XML

Installation

Important News - November 2007

These installation directions are specific to the "pre-1.0" branch of xml-qstat. Major changes have recently been committed to the SVN trunk that obsolete these instructions. Make sure you are usinng xml-qstat_pre-1.0 if you want these instructions to make sense!

The new version of xml-qstat, as represented by the files in the SVN trunk/ branch is now pure-Java. No additional perl helper daemons or perl CGI's are required! Installation and configuration is much simpler. This is all thanks to Petr Jung who contributed Java CommandGenerator classes that allow our web application to directly query Grid Engine for XML status data.

The main reason we are still offering the "pre-1.0" install directions and the downloadable tarball is that we have not rewritten the documentation to reflect the new features. If you are interested in being a beta tester of the "new" pure-Java xml-qstat, please visit this wiki page: http://wiki.gridengine.info/wiki/index.php/Xml-qstat-1.0 until we package up an 'official' release, we'll use the Wiki to collaborate on documentation and hints for the new version.

Package Overview

The choice of installation directory is up to you, we recommend organizing the files in the following manner:

$ ls -l /opt/xml-qstat/
drwxr-xr-x    cocoon-2.1.8
drwxrwxr-x    httpi
drwxrwxr-x    scripts
drwxrwxr-x    xmlqstat

The cocoon directory is a standard Apache Cocoon installation. The only file touched inside the cocoon folder is a customized "mount-table.xml" configuration file that informs the application server that the URI-prefix of "/xmlqstat/" should be handled by the "xmlqstat/" folder.

The httpi directory is optional and contains a tiny, fast and secure pure-Perl webserver known as httpi. This webserver is used to run a trivial CGI application that serves as a dynamic XML generator for cocoon when the cached data is not suitable. The primary use so far is to service requests for job related grid status data. When a user clicks on a Job ID within the web interface, cocoon will call upon httpi to run and return the raw XML results from the "qstat -j -xml" query. Eventually this functionality will be replaced with a pure-Java "Cocoon Generator" that performs the same task without requiring an external additional webserver.

For users with administrative access to an existing Apache webserver, httpi should not be needed at all. The xml-qstat scripts/ directory will contain simple perl CGI's capable of feeding XML job-status data to the Cocoon application server.

The scripts folder currently contains 2 perl scripts. The "sge-xml-cacher.pl" script is a Perl daemon script that runs constantly and periodically refreshes the cached Grid Engine XML status data. This cached data is what cocoon uses to generate various views and interfaces. The default setting for this daemon is to refresh Grid Data every 30 seconds (this value is user configurable). The perl daemon is used so that Cocoon does not call "qstat -f -xml" every time a user hits the web page. The caching mechanism greatly reduces the load on the SGE qmaster and the Cocoon application server. The second perl script, "create-threshold-config.pl" is optional and is a helper script that can query Grid Engine to learn the configured load alarm thresholds for each queue. Since Grid Engine only reports current load values (and not the configured alarm thresholds) this helper script can create a secondary configuration file that is read by Cocoon to generate customized graphical "load ratio" reports within the web interface.

The xmlqstat folder is where this application lives. It is organized into the following subfolders:

$ ls -l xmlqstat/
drwxrwxr-x   css
drwxrwxr-x   images
drwxrwxr-x   javascript
-rw-rw-r--   sitemap.xmap
drwxrwxr-x   xml
drwxrwxr-x   xsl

 

The css folder contains CSS stylesheets used for separation of content from presentation. The image folder contains images, icons and graphic elements. The javascript folder contains javascript used for presentation and display preference formatting. The sitemap.xmap is a Cocoon configuration file that tells the application server how to handle, route and process various transformation requests. The xml folder contains user-customizable configuration files and the raw XML source for the documentation and help files. The xsl folder contains the XSLT stylesheets that guide the transformation of Grid Engine XML into various output forms.

Generally speaking, the only file a user will have to edit is the xmlqstat/xml/CONFIG.xml. This file is used to set some simple configuration preferences. In some cases, a user may have to edit sitemap.xmap to make minor configuration changes (such as specifying a different default URL and port number for the optional httpi webserver).

Installation

1. Make sure you have Perl and Java available and that the $JAVA_HOME environment variable is set.
2. Create a folder to hold the misc. components (something like xml-qstat/)
3. Download Apache Cocoon version 2.1.8 or later. Simply unpacking the distribution into the xml-qstat/ folder and running the "build.sh" script should work fine. The build may take several minutes and when it completes you will have a script called "cocoon.sh" that can be run. In its default form (with 2.1.8 or later) simply running "./cocoon.sh" as a non-privledged user will cause Cocoon to start up its own internal application server that binds to all available network interfaces and listens for incoming connections at port number 8888. To test, simply point your web browser at the http://(your servername):8888/ -- if it works you will see a Cocoon welcome screen. Firewall users should make sure that TCP connections to port 8888 are allowed.
4. Download the xmlqstat.tar.gz distribution, when unpacked into the xml-qstat/ folder it will populate the scripts/ and xmlqstat/ folders.
5. optional: Download httpi into the xml-qstat/ folder. This is only necessary if you want to enable the feature that allows job-level Grid Engine data to be reported (clicking on a JobID will show information about the running or pending job).
6. Create the mount-table.xml configuration file in the cocoon directory (details are below).
7. Edit the xmlqstat/xml/CONFIG.xml file to set local configuration parameters (details are below)
8. Edit the scripts/sge-xml-cacher.pl script to configure the full Unix file path to where cached SGE status data will be stored. If data is cached to a file named "cached-sge-data.xml" within the xmlqstat/xml/ folder then xmlqstat/sitemap.xmap can be left unchanged. If the cached data is stored elsewhere, the sitemap.xmap file located in xmlqstat/ will have to be edited in several locations to let cocoon know where the raw XML source resides.
9. Start caching grid data! If your SGE environment has been set up (you can run 'qstat' via the command line) then starting the perl caching daemon should be as simple as entering the scripts/ folder and issuing the command "./sge-xml-cacher.pl -d". To stop the daemon, issue the command "./sge-xml-cacher.pl -k". To verify that the daemon is working, check the process table to see if it exists and then monitor the timestamps on the cache file. In its default setting, XML data should be updated every 30 seconds.
10. The web interface should be ready for testing. Point a web browser at the URL http://(your server name):8888/xmlqstat/qstat.html

Advanced Installation: Cocoon behind Apache Proxies

Many Unix servers already have an Apache webserver running on on the system. If you have administrative access to this server you can proxy the Cocoon application server behind Apache. The primary benefit is removing the browser requirement to connect to TCP port 8888. The URL changes from http://(your server name):8888/xmlqstat/qstat.html to http://(your server name)/xmlqstat/qstat.html. This also allows firewall administrators to seal port 8888 closed against all connections not coming from the apache server. The Apache configuration directives to do this would be:

## Let Apache proxy connections to cocoon on port 8888 forward and reverse...
ProxyPass /xmlqstat http://127.0.0.1:8888/xmlqstat
ProxyPassReverse /xmlqstat http://127.0.0.1:8888/xmlqstat

Creating mount-table.xml

Within the cocoon directory it is possible to create a file called mount-table.xml. The purpose of this file is to allow webapp folders to exist elsewhere on the system rather than forcing them to be installed inside the cocoon/ folder itself. If your file layout matches the example:

$ ls -l /opt/xml-qstat/
drwxr-xr-x    cocoon-2.1.8
drwxrwxr-x    httpi
drwxrwxr-x    scripts
drwxrwxr-x    xmlqstat

Then you would simply create a file called cocoon-2.1.8/mount-table.xml with contents that looked like this (yes the relative uri-prefix "../../../xmlqstat/" path is correct for the directory layout shown above) :

  <mount-table>
 
    <!-- Map the browser URI "/xmlqstat" to our webapp folder -->
    <mount uri-prefix="xmlqstat/" src="../../../xmlqstat/"/>
    
  </mount-table>

After creating this file, it will be necessary to stop and restart Cocoon if it was running previously. Once cocoon is restarted, the application server will know how to find the xmlqstat/ application files when web clients request the URI-prefix of "/xmlqstat".

Customizing CONFIG.xml

Within the xmlqstat/xml/ directory there is a file called CONFIG.xml that needs to be edited to reflect the local site installation. The file is well documented internally so it should be a simple process to edit. The following configuration questions are asked:

1. compat_mode. Starting with the release of Grid Engine 6.0u7 there have been major XML output changes made to the codebase. This configuration option should be set to "yes" if you are using a version of Grid Engine older than version 6.0u7. Set it to "no" if you are using 6.0u7 or later.
2. use_alarm-threshold_data. This is an optional feature. If you run the helper script located in scripts/create-threshold-config.pl, a special helper configuration XML file will be created that lists the Grid Engine load alarm threshold values for each queue instance. These values can then be compared to the real-time load data reported by Grid Engine. When enabled, this feature will show a graphical "Load Ratio" column in the web interface. When the value is set to "no", the Load Ratio column data is not displayed.
3. UTC_timezone_offset. This is only used for Atom 1.0 Feeds which mandate RFC-3339 formatted timestamps. Configuring this value allows the user to adjust for local timezones. The RFC3339 timestamp uses UTC time which can be offset to match a local timezone.
4. baseURL This is also only used for Atom 1.0 Feeds so they can contain a full, explict http:// URL pointing back to the xmlqstat web interface. All other portions of the web application use relative URLs. Configure this with the fully qualified URL to your installation.