Wednesday, September 28, 2011

Passing plain Java objects to R using RCaller

Well, you are using RCaller for your statistical calculations. Probably, you are passing your double arrays to R and type some R commands in order to get the desired outputs. After a calculation process, you handle the returned arrays through the parser. This is the general use of RCaller.

Suppose that you have got a Java class which has got some variables with data types int, short, long, float, double and String. This class also includes some arrays of types int[], double[], ..., String[]. Of course it may include some functions, constructors or anything else. But we don't care about this for now.  How about passing this class with its publicly defined variables to R? Yeah! It is possible in its last submitted revision.

Have a look at the Java class below:


class TestClass {

  public int i = 9;
  public float f = 10.0f;
  public double d = 3.14;
  public boolean b = true;
  public String s = "test";
}

This class simply includes five publicly defined variables with basic data types. Our other class inherits the TestClass and defines some additional arrays:


class TestClassWithArrays extends TestClass {

  public int[] ia = new int[]{1, 2, 3, 4, 5};
  public double[] da = new double[]{1.0, 2.0, 3.0, 4.0, 9.9, 10.1};
  public String[] sa = new String[]{"One", "Two", "Three"};
  public boolean[] ba = new boolean[]{true, true, false};
}

Ok, they are very simple but there is no reason those classes not to have any methods. Whatever those classes have methods, we consider them as data structures.

Lets pass this data structure to R:


TestClassWithArrays tcwa = new TestClassWithArrays();
    JavaObject jo = new JavaObject("tcwa", tcwa);

    RCaller rcaller = new RCaller();
    rcaller.setRscriptExecutable("/usr/bin/Rscript");
    rcaller.cleanRCode();

    rcaller.addRCode(jo.produceRCode());
    rcaller.runAndReturnResult("tcwa");


Well, if there is no expection we have the results in a R list named "tcwa". This R list includes all of the elements that included in TestClassWithArrays and TestClass with their values.

This is an example of proof, the related @Test method is ready for browsing in the Google Code:

@Test
  public void TestClassWithArrays() throws IllegalAccessException, IOException {
    TestClassWithArrays tcwa = new TestClassWithArrays();
    JavaObject jo = new JavaObject("tcwa", tcwa);

    RCaller rcaller = new RCaller();
    rcaller.setRscriptExecutable("/usr/bin/Rscript");
    rcaller.cleanRCode();

    rcaller.addRCode(jo.produceRCode());
    rcaller.runAndReturnResult("tcwa");

    int[] expectedIntArray = rcaller.getParser().getAsIntArray("ia");
    for (int i = 0; i < tcwa.ia.length; i++) {
      assertEquals(expectedIntArray[i], tcwa.ia[i]);
    }


    double[] expectedDoubleArray = rcaller.getParser().getAsDoubleArray("da");
    for (int i = 0; i < tcwa.da.length; i++) {
      assertEquals(expectedDoubleArray[i], tcwa.da[i], delta);
    }

    String[] expectedStringArray = rcaller.getParser().getAsStringArray("sa");
    for (int i = 0; i < tcwa.sa.length; i++) {
      assertEquals(expectedStringArray[i], tcwa.sa[i]);
    }

  }

It is shown that in examples, in R side, we can access to elements with their original names that defined in the Java class. That sounds good.

Finally, we can pass our Java objects with defined contents. This use of RCaller narrows the code of addDoubleArray, addIntArray and reduce all of them to simple command of

 JavaObject jo = new JavaObject("tcwa", tcwa);
.
.
.
rcaller.addRCode ( jo.produceRCode() );
 

It is simplicity...

Friday, September 23, 2011

How to follow up temperature of system room,

able to visit following link For more example and with different devices like hp, cisco, netscreen..;
http://stdioe.blogspot.com/2011/10/checking-system-room-temperature-with.html
also should be check http://stdioe.blogspot.com/2011/11/nokia-application-for-system-room.html article. This article contain a mobile phone client application to follow up to system room temperature..


I needed to find a solution for the question above for the company which I'm working for. I wanna talk about my quick solution in this article.

There are several IMB Blade chassis, cisco routers, Juniper devices, HP / IBM rock mountable servers, Cisco/Hp/Nortel swithces etc. in my system room.. Actually, there are many alternatives but I selected the IBM Blade chassis to read data of temperature. Because, It contains a lot of blade servers and they are really business critical. Somehow that blade chassis should always be powered on..

I have a Linux server for monitoring the network and something like that. It has got Apache Http Server, MySQL server, a Perl interpreter, a Php interpreter installed. I created a DB and a table to record temperature values. I wrote a script to connect to IBM blade chassis and get temperature value. This script gets the values and record them to a text file;

Read data from IBM Blade Chassis (read.pl);

#!/usr/bin/perl
use Net::Telnet ();
$t = new Net::Telnet (Timeout => 10);
$t->open("x.x.x.x");
$t->login("USERID", "password");
$t->cmd("env -T system:mt[1]");
@lines = $t->cmd("temps");
#print @lines;

open (HEDEF, ">/root/ibmTemperature/lastTemp.txt") || die ("Could not open file
$!");
print HEDEF "@lines";
close (HEDEF);


recorded data to the lastTemp.txt file is like this;

              Warning
Value Warning Reset
----- ------- -------
25.00 38.00 33.00
...
..
.


I need the first value; the value of 25.00 at line 1. Use following command to eleminate unwanted parts.

echo `/usr/bin/head -4 /root/ibmTemperature/lastTemp.txt | /usr/bin/tail -1 | /usr/bin/gawk -F ' ' ' {print $1} '`> /root/ibmTemperature/tempfile.txt

Write data to MySQL table (write.php);

I decide to use different languages for each step. The following php script reads the data from blade chassis, writes the data to mysql database server

#!/usr/bin/php


I did set to execution permission to the script given above. I also wrote the interpreter path to head of the script source. So therefore, I can execute it directly. I also have to say, I almost always use the full path names in all scripts. Because, if I use relative paths in the scripts, I need login on the treminal / console to execute it to get $PATH variable value from environmental values structure. But, If I use full path in my scripts, I can execute it from everywhere (for example, execute from crontab)

shell script for execute all (run.sh),

#!/bin/bash
/root/ibmTemperature/read.pl
echo `/usr/bin/head -4 /root/ibmTemperature/lastTemp.txt | /usr/bin/tail -1 | /usr/bin/gawk -F ' ' ' {print $1} '`> /root/ibmTemperature/tempfile.txt
/root/ibmTemperature/write.php



Now, I can add run.sh script in the crontab for executing it once a minute and run.sh script to be able to trigger the read.pl, write.php and some linux commands. And now, I have to explain that linux commands;

a) The head is captured from first 4 lines of lastTemp.txt text file. The pipe captures an output from that head and inputs to tail command. So tail command captures the last line from the output of head command and next pipe creates an input for gawk..

This structure is being executed once a minute for adding the current temperature information to MySQL table with mktime date/time format. The following php script creates a clear output for users on a web-page. (my webpage hosted on /systemRoom/ directory in web root directory.)


 









System room temperature graphic (Last 24 Hours):";
echo "";
echo "";
echo "\n";
$k = 0;
while($line = mysql_fetch_array($result)) {
$current = (3*$line["temp"]);
$current_temp = $line["temp"];

if($current < "72") { $color = "lightblue.PNG"; } elseif($current < "85") { $color = "blue.PNG"; } else { $color = "red.PNG"; } if($k == 0) { $now = $line["date"]; } else { if(strlen($line["date"]) > 5) {
$end = $line["date"];
}
}

echo "\n";
}
echo "";
//echo "";
echo "

";
echo "";
echo "";
//echo "Now (0 point on graphic): ".date("Y/m/d - H:i:s", $now)."
Oldest (End point on graphic): ".date("Y/m/d - H:i:s", $end)."
";
?>


";
echo "";
echo "





: Recommended Values Range

: Acceptable Values Range

: Dangerous Values Range

Thursday, September 22, 2011

How to Connect Juniper Netscreen Device using Perl Scripts

The site http://search.cpan.org contains a usable and easy library to connect netscreen devices. Name of the library is Net::Telnet::Netscreen. Its use is straightforward:

#!/usr/bin/perl
use Net::Telnet ();
$t = new Net::Telnet (Timeout => 10,
Prompt => '/ns5gt-adsl->/');

$t->open("IP_ADDR_of_NetscreenDevice");
$t->login(netscreenUsername, Password);
@lines = $t->cmd("get policy id X");
print @lines;
$t->cmd("exit");
$t->cmd("exit");


IP_ADDR_of_NetscreenDevice, netscreenUsername and Password expressions can be replaced with an address, an username and a password, respectively.

How to Connect Cisco Router with Perl Scripts

( Click for PHP version: http://stdioe.blogspot.com/2011/11/how-to-connect-cisco-router-with-php.html )

First of all, I have to explain how to configure the Cisco router for telnet connectivity. Because, the Cisco router supports the telnet password and privilege password, It also supports username and password combination for logging in. So there are two different type to logging in.

The following explanation of Cisco router configurations are from stratch. So we have to connect to router via console cable (rollover cable) and serial port on computer and terminal application. If you use MS Windows operating system, you can use Hyper terminal or different third party terminal applications. If you use Linux operating system, you have several choices. I usually use the minicom in my personel use laptop . But the problem is that It hasn't got any serial ports. The solution is to use the usb to serial converter adapter with requisite drivers installed in my Linux.

Router-A Configuration:
Router> Enable
Router# configure Terminal
Enter configuration commands, one per line. End with CNTL/Z.
Router(config)# enable secret 0 cisco
Router(config)# line vty 0 4
Router(config-line)#login
% Login disabled on line 6, until 'password' is set
% Login disabled on line 7, until 'password' is set
% Login disabled on line 8, until 'password' is set
% Login disabled on line 9, until 'password' is set
% Login disabled on line 10, until 'password' is set
Router(config-line)#password cisco
00:00:54: %SYS-5-CONFIG_I: Configured from console by console
Router(config-line)# ^Z
Router#write memory

The following perl script to connect to Router-A without AAA;

#!/usr/bin/perl

use Net::Telnet::Cisco;
my $session = Net::Telnet::Cisco->new(Host => 'x.x.x.x');
$session->login('', 'password');

# Execute a command
my @output = $session->cmd('show version');
print @output;

# Enable mode
if ($session->enable("enable_password") ) {
@output = $session->cmd('show privilege');
print "My privileges: @output\n";
} else {
warn "Can't enable: " . $session->errmsg;
}

$session->close;

After than write this perl script above, of course you have to add execution permission to script file. For example, if the file name of this script is sample.pl then simply type

chmod +x sample.pl

Router-B Configuration (Updating to AAA model):

Router(config)#aaa new-model
Router(config)#username TelnetUser privilege 15 password 0 TelnetPassword

The following perl script to connect to Router-B;

The difference between the first sample and the second sample is that,
first router configuration is done with telnet password and password.

Anyway you can use the Net::Telnet::Cisco Library which is written in Perl. If you are using a Linux Distro, probably your package manager already contains it.

#!/usr/bin/perl

use Net::Telnet::Cisco;

my $session = Net::Telnet::Cisco->new(Host => 'x.x.x.x');
$session->login('TelnetUser', 'TelnetPassword');

# Execute a command
my @output = $session->cmd('show version');
print @output;

# Enable mode
@output = $session-> cmd('show privilege');
print "My privileges: @output\n";
$session->close;

If you want to add "Net::Telnet:Cisco" or something like that manually, you can search the related perl library on site http://search.cpan.org.
For Example http://search.cpan.org/~joshua/Net-Telnet-Cisco-1.10/Cisco.pm link is used in the sample we have just given.
And you can also download http://search.cpan.org/CPAN/authors/id/J/JO/JOSHUA/Net-Telnet-Cisco-1.10.tar.gz compressed file.

Note:
After extracting it, enter extracted directory. Execute perl Makefile.PL.
The "make" and "make install" commands produces the output below:

user@hostn:~/DIR> tar xvfz Net-Telnet-Cisco-1.10.tar.gz 
Net-Telnet-Cisco-1.10/
Net-Telnet-Cisco-1.10/README
Net-Telnet-Cisco-1.10/Cisco.pm
Net-Telnet-Cisco-1.10/.cvsignore
Net-Telnet-Cisco-1.10/MANIFEST
Net-Telnet-Cisco-1.10/test.pl
Net-Telnet-Cisco-1.10/MANIFEST.SKIP
Net-Telnet-Cisco-1.10/Changes
Net-Telnet-Cisco-1.10/INSTALL
Net-Telnet-Cisco-1.10/Makefile.PL
Net-Telnet-Cisco-1.10/TODO
user@hostn:~/DIR> cd Net-Telnet-Cisco-1.10/
user@hostn:~/DIR/Net-Telnet-Cisco-1.10> perl Makefile.PL

Checking if your kit is complete...
Looks good
Writing Makefile for Net::Telnet::Cisco
user@hostn:~/DIR/Net-Telnet-Cisco-1.10> make
cp Cisco.pm blib/lib/Net/Telnet/Cisco.pm
AutoSplitting blib/lib/Net/Telnet/Cisco.pm (blib/lib/auto/Net/Telnet/Cisco)
Manifying blib/man3/Net::Telnet::Cisco.3pm
user@hostn:~/DIR/Net-Telnet-Cisco-1.10> make install
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
ERROR: Can't create '/usr/lib/perl5/site_perl/5.12.3/Net/Telnet'
Do not have write permissions on '/usr/lib/perl5/site_perl/5.12.3/Net/Telnet'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
at -e line 1
make: *** [pure_site_install] Error 13
user@hostn:~/DIR/Net-Telnet-Cisco-1.10> sudo make install
root's password:
Appending installation info to /usr/lib/perl5/5.12.3/i586-linux-thread-multi/perllocal.pod
user@hostn:~/DIR/Net-Telnet-Cisco-1.10>

The last step is installing which is required root permissions. So When used without root permission, It returned an error than used "sudo" to get root permission, It finally successful.

Saturday, September 17, 2011

RCaller: Support for sequential commands with a single process

I think, this revision will be the foundation of the version  2.1. RCaller is supposed to be slow but the easiest way of calling R from Java.

Finally I have implemented the method runAndReturnResultOnline() for running sequential commands in a single process. What does this stand for? Let me give an example to explain this:

Suppose that you want to perform a simulation study to measure the success of your new procedure. For this, you decide to draw random numbers from a distribution and calculate something and handle the results in Java. RCaller creates  Rscript processes for each single iteration. This cause to too many operating system calls.

Latest release of RCaller includes the method for this. Lets have a look at the Test file:


@Test
  public void onlineCalculationTest() {
    RCaller rcaller = new RCaller();
    rcaller.setRExecutable("/usr/bin/R");
    rcaller.cleanRCode();
    rcaller.addRCode("a<-1:10");
    rcaller.runAndReturnResultOnline("a");
    assertEquals(rcaller.getParser().getAsIntArray("a")[0], 1);

    rcaller.cleanRCode();
    rcaller.addRCode("b<-1:10");
    rcaller.addRCode("m<-mean(b)");
    rcaller.runAndReturnResultOnline("m");
    assertEquals(rcaller.getParser().getAsDoubleArray("m")[0], 5.5, 0.000001);

    rcaller.cleanRCode();
    rcaller.addRCode("a<-1:99");
    rcaller.addRCode("k<-median(a)");
    rcaller.runAndReturnResultOnline("k");
    assertEquals(rcaller.getParser().getAsDoubleArray("k")[0], 50.0, 0.000001);
  }
  }
 

In first stage,we are creating an integer vector and getting the first element. In the second one, we are creating the same integer vector with a different name and calculating the arithmetic mean. In the last one, we are recreating the vector a and getting the median, which is equal to 50.

This example uses the same RCaller object. In first stage, the R executable file (it is /usr/bin/R in my Ubuntu Linux) is created once. In second stage the same R file is used and no longer processes are created again. In this stage, the vector a is accessible and still remains alive. At the last stage, b is alive again and a is recreated. So this example does not cause the R to open and close three times but only once.

This modification speeds up the RCaller, but it can be still considered as slow.
However, it is still easy to implement and much more faster than the previous implementation.

Have Fun!