About History Screenshots Demo Requirements Download Install MailList Acknowledgments XML

Installation

Important News

No officially packaged release exists yet. The instructions below are temporary instructions for advanced users, people who really want to see what the new github hosted code can do as well as for people who are interested in becoming developers or contributors.

In particular the git instructions deal with a temporary branch called "chrisdag-dev-branch" containing a cleaned up set of files and folders where old and unused java and netbeans related project files have been deleted. If these changes work out, we'll merge the branch back with the master branch and update these instructions accordingly.

Package Overview

The choice of installation directory is up to you, we recommend organizing the various components in the following manner:

$ ls -l /opt/xml-qstat/
drwxr-xr-x    cocoon-2.1.11
drwxrwxr-x    httpi
drwxrwxr-x    xml-qstat

 

The cocoon directory is a standard Apache Cocoon installation. The only file touched inside the cocoon folder is a customized "mount-table.xml" configuration file that informs the application server that the URI-prefix of "/xmlqstat/" should be handled by the "xmlqstat/" folder.

The httpi directory is optional and contains a tiny, fast and secure pure-Perl webserver known as httpi. This webserver is used to run a trivial CGI application that serves as a dynamic XML generator for cocoon when the cached data is not suitable. The primary use so far is to service requests for job related grid status data. When a user clicks on a Job ID within the web interface, cocoon either use internal methods to call "qstat -j -xml" directly or it can call upon httpi and small CGI scripts hosted within. Since the current version of xml-qstat can directly pull job status data from Grid Engine the use of httpi is for more advanced cases where other more customized sources of raw XML data may be desired.

Inside xml-qstat/

$ ls -l xml-qstat/
drwxrwxr-x    init-scripts
drwxrwxr-x    scripts
drwxrwxr-x    xmlqstat

 

The init-scripts/ folder contains helper scripts for automatically running or daemonizing programs such as the cocoon application server and the XML caching scripts.

The scripts folder currently contains 2 perl scripts, a CGI script called 'jobinfo' and a "qstat.sh" shell script that the application can use to directly query the Grid Engine qmaster. The "sge-xml-cacher.pl" script is a Perl daemon script that runs constantly and periodically refreshes the cached Grid Engine XML status data. This cached data is what cocoon uses to generate various views and interfaces. The default setting for this daemon is to refresh Grid Data every 30 seconds (this value is user configurable). The perl daemon is used so that Cocoon does not call "qstat -f -xml" every time a user hits the web page. The caching mechanism greatly reduces the load on the SGE qmaster and the Cocoon application server. The second perl script, "create-threshold-config.pl" is optional and is a helper script that can query Grid Engine to learn the configured load alarm thresholds for each queue. Since Grid Engine only reports current load values (and not the configured alarm thresholds) this helper script can create a secondary configuration file that is read by Cocoon to generate customized graphical "load ratio" reports within the web interface. The 'jobinfo' script can be used as a CGI under Apache or httpi if an external URL-based source for SGE XML status data is desired.

The xmlqstat folder is where this application lives. It is organized into the following subfolders:

$ ls -l xmlqstat/
drwxrwxr-x   config
drwxrwxr-x   css
drwxrwxr-x   images
drwxrwxr-x   javascript
-rw-rw-r--   sitemap.xmap
drwxrwxr-x   xml
drwxrwxr-x   xsl

The config folder contains XML configuration files for the xml-qstat application. The css folder contains CSS stylesheets used for separation of content from presentation. The image folder contains images, icons and graphic elements. The javascript folder contains javascript used for presentation and display preference formatting. The sitemap.xmap is a Cocoon configuration file that tells the application server how to handle, route and process various transformation requests. The xml folder contains the raw XML source for documentation and help files. The xsl folder contains the XSLT stylesheets that guide the transformation of Grid Engine XML into various output forms.

Generally speaking, the only file a user will have to edit are the files in the config/ directory. In some cases, a user may have to edit sitemap.xmap to make minor configuration changes (such as specifying a different default URL and port number for the optional httpi webserver).

It helps to be familiar with the contents of sitemap.xmap as this is the single cocoon configuration file that controls what XML and XSLT transformations occur when a particular URL path is seen.

Installation Overview

1. Grid Engine 6.2 should be installed and functioning. Earlier versions of Grid Engine produce different XML output and may not be supported well.
2. Make sure you have Perl and Java available and that the $JAVA_HOME environment variable is set.
3. Create a folder to hold the misc. components (something like xml-qstat/)
4. Download Apache Cocoon version 2.1, preferably the latest stable version 2.1.11. Apache Cocoon 2.2 (any version) is not supported at all as Cocoon 2.2 represents a radical change from 2.1. Simply unpacking the distribution into the xml-qstat/ folder and running the "build.sh" script should work fine. The build may take several minutes and when it completes you will have a script called "cocoon.sh" that can be run. In its default form (with 2.1.8 or later) simply running "./cocoon.sh" as a non-privledged user will cause Cocoon to start up its own internal application server that binds to all available network interfaces and listens for incoming connections at port number 8888. To test, simply point your web browser at the http://(your servername):8888/ -- if it works you will see a Cocoon welcome screen. Firewall users should make sure that TCP connections to port 8888 are allowed.
5. Checkout or download xml-qstat from github.com, detailed instructions are outlined below.
6. optional: Download httpi into the xml-qstat/ folder.
7. Create the mount-table.xml configuration file in the cocoon directory (details are below).
8. Build xml-qstat.jar and install it into the proper Cocoon lib/ direcory (details are below)
9. Edit the xmlqstat/config/config.xml file to set local configuration parameters (details are below)
10. Edit the scripts/sge-xml-cacher.pl script to configure the full Unix file path to where cached SGE status data will be stored. If data is cached to a file named "cached-sge-data.xml" within the xmlqstat/xml/ folder then xmlqstat/sitemap.xmap can be left unchanged. If the cached data is stored elsewhere, the sitemap.xmap file located in xmlqstat/ will have to be edited in several locations to let cocoon know where the raw XML source resides.
11. Start caching grid data! If your SGE environment has been set up (you can run 'qstat' via the command line) then starting the perl caching daemon should be as simple as entering the scripts/ folder and issuing the command "./sge-xml-cacher.pl -d". To stop the daemon, issue the command "./sge-xml-cacher.pl -k". To verify that the daemon is working, check the process table to see if it exists and then monitor the timestamps on the cache file. In its default setting, XML data should be updated every 30 seconds.
12. The web interface should be ready for testing. Point a web browser at the URL http://(your server name):8888/xmlqstat/qstat.html

 

Installation Details: Apache Cocoon

Only Apache Cocoon 2.1 is supported, the 2.2 release will not work. With $JAVA_HOME set, the application should build itself via the "build.sh" script. After it is built the "cocoon.sh" script will launch the application server. Successful operation can be verified by pointing a web browser at http://<host>:8888/ and seeing the "Welcome to Apache Cocoon" message.

When starting cocoon.sh to service the xml-qstat web application, the following conditions must be met:

- The SGE settings and environment variables must be present (source the Grid Engine settings.sh or settings.csh script before starting the cocoon server)

- The xml-qstat scripts/qstat.sh and scripts/qhost.sh must be in the $PATH before starting the cocoon server so that the server can find and execute them when needed

These path and environment settings can be adjusted manually before starting the server or can be edited into the cocoon.sh script.

Advanced Installation: Cocoon behind Apache Proxies

Many Unix servers already have an Apache webserver running on on the system. If you have administrative access to this server you can proxy the Cocoon application server behind Apache. The primary benefit is removing the browser requirement to connect to TCP port 8888. The URL changes from http://(your server name):8888/xmlqstat/qstat.html to http://(your server name)/xmlqstat/qstat.html. This also allows firewall administrators to seal port 8888 closed against all connections not coming from the apache server. The Apache configuration directives to do this would be:

## Let Apache proxy connections to cocoon on port 8888 forward and reverse...
ProxyPass /xmlqstat http://127.0.0.1:8888/xmlqstat
ProxyPassReverse /xmlqstat http://127.0.0.1:8888/xmlqstat

Installation: Obtaining xml-qstat from github.com

Basic 'git' instructions can be found on the download page.

If you do not have a local 'git' client installed you can surf directly to http://github.com/bioteam/xml-qstat/tree/chrisdag-dev-branch and click on the "download" link.

 

Building xml-qstat.jar and installing it into Cocoon

When operating with live (non-cached) data, the web application needs a method of calling SGE programs such as 'qstat' and 'qhost' in order to fetch XML data. This is done with a Java based helper class that must be built into a .jar file and installed in a place where the Cocoon application server can access it. At the top level of the xml-qstat checkout will be a shell script called "build-xmlqstat-jar.sh".

To build and install the Jar file:

1. Edit build-xmlqstat-jar.sh to set the cocoonBase directory

2. Make sure Java and 'javac' is in your path

3. Run the script

4. Copy the resulting build/xml-qstat.jar file into the location specified by the build script output

5. Stop and restart Cocoon (if necessary)

 

Creating mount-table.xml

Within the cocoon directory it is possible to create a file called mount-table.xml. The purpose of this file is to allow webapp folders to exist elsewhere on the system rather than forcing them to be installed inside the cocoon/ folder itself. If your file layout matches the example:

$ ls -l /opt/xml-qstat/
drwxr-xr-x    cocoon-2.1.11
drwxrwxr-x    httpi
drwxrwxr-x    xml-qstat

Then you would simply create a file called cocoon-2.1.11/mount-table.xml with contents that looked like this (yes the relative uri-prefix "../../../xml-qstat/xmlqstat/" path is correct for the directory layout shown above) :

  <mount-table>
 
    <!-- Map the browser URI "/xmlqstat" to our webapp folder -->
    <mount uri-prefix="xmlqstat/" src="../../../xml-qstat/xmlqstat/"/>
    
  </mount-table>

After creating this file, it will be necessary to stop and restart Cocoon if it was running previously. Once cocoon is restarted, the application server will know how to find the xmlqstat/ application files when web clients request the URI-prefix of "/xmlqstat". Remember the other prerequisites including the SGE environment variables and the $PATH adjustment so that xml-qstat/scripts/qstat.sh can be found.

Customizing config/config.xml

Within the xmlqstat/config/ directory there is a file called config.xml that needs to be edited to reflect the local site installation. The file is well documented internally so it should be a simple process to edit.

Using Cached Grid Engine XML

To reduce load on the qmaster and avoid calling 'qstat' each time a page is accessed, cached SGE data can be used. A script in the scripts/ folder called "xmlqstat-cacher.pl" can perform this function.

The cache location is defined by how the cluster is named in config/config.xml. Create a directory with "cache-" prepended to the cluster name within the xmlqstat folder (at the same level as where the sitemap.xmap file exists). The application will automatically detect the presence of the cache directory if this naming and location convention is followed.

The xmlqstat-cacher.pl program will create the needed cache files "qstat.xml", "qstatf.xml" and "qhost.xml". If you run into problems with a web application error mentioning "qlicserver.xml" then copy the dummy qlicserver.xml_dummy file from the config/ folder into the cache directory as "qlicserver.xml".