MySVC

Table of Contents

Introduction

MySVC (My Service) is an open source UNIX framework for shell script based web services provider.

This framework let you easily realize fast prototypes of your web services using standard UNIX (bourne) shell scripts as implementation for web service operations.

It is an engineering of a previous framework that i have developed years ago as employee, from startup to shutdown, of the great blu mobile telephone operator to handle prepaid, postpaid and VPN provisioning interfaces and workflows on all blu network elements, such as HLR, VMS and Intelligent Network, including then other ideas working on CRM, mediation devices, middleware systems and XML technologies such as XSLT and DocBook.

Free use of these software is granted under the terms of GNU General Public Licence (GPL). See License.

Now are released the following standalone tools part of MySVC framework:

MySTE (My Service Template Engine)
MySDE (My Service Development Environment)
MySPL (My Service Protocol Library)
MyExpr (My Service Expression Engine)

MySTE

MySTE (My Service Template Engine) is a template engine, useful in an ICT environment.

It could be used for example:

as filter copying from stdin to stdout where input can contains directives (identified by lines beginning with "@"), such as parameter definitions, and normal text referencing those parameters, e.g.:

 ...
 login is @username@/@password@...
 ...

as tool to handle configuration files from shell script, e.g.:

 ...
 usr=`myste myconfigfile username`
 pwd=`myste myconfigfile password`
 ...

to generate provisioning commands for network elements of a mobile operator, e.g.:

input

 @define sim_activate_prepaid(msisdn,imsi)
 SIM:ACTIVATE:PREPAID:@msisdn[3]@:@imsi@:TIMESTAMP:@$(date '+%Y%m%d%H%M%S')@
 @end
 @define sim_list_postpaid
 222981122334455 393801234567
 222981122334456 393801234568
 222981122334457 393801234569
 @end
 @define sim_list_prepaid
 393801234567 222981122334455
 393801234568 222981122334456
 393801234569 222981122334457
 @end
 @foreach sim_list_postpaid
 SIM:ACTIVATE:POSTPAID:@2[3]@:@1@:TIMESTAMP:@$(date '+%Y%m%d%H%M%S')@
 @end
 @foreach sim_list_prepaid
 @sim_activate_prepaid(@1@,@2@)@
 @end

output

 SIM:ACTIVATE:POSTPAID:3801234567:222981122334455:TIMESTAMP:20090406231917
 SIM:ACTIVATE:POSTPAID:3801234568:222981122334456:TIMESTAMP:20090406231917
 SIM:ACTIVATE:POSTPAID:3801234569:222981122334457:TIMESTAMP:20090406231917
 SIM:ACTIVATE:PREPAID:3801234567:222981122334455:TIMESTAMP:20090406231917
 SIM:ACTIVATE:PREPAID:3801234568:222981122334456:TIMESTAMP:20090406231917
 SIM:ACTIVATE:PREPAID:3801234569:222981122334457:TIMESTAMP:20090406231917

to handle ftp put get del dir from command line with or without recursion on subdirectories, e.g.:

 @dir ftp://myusername:mypassword@myremotehost/my/remote/file
 @dir ftp://myusername:mypassword@myremotehost/my/remote/dir *.txt
 @get ftp://myusername:mypassword@myremotehost/my/remote/file /my/local/dir
 @get ftp://myusername:mypassword@myremotehost/my/remote/dir /my/local/dir *.txt
 @put ftp://myusername:mypassword@myremotehost/my/remote/file /my/local/dir
 @put ftp://myusername:mypassword@myremotehost/my/remote/dir /my/local/dir *.txt
 @del ftp://myusername:mypassword@myremotehost/my/remote/file
 @del ftp://myusername:mypassword@myremotehost/my/remote/dir

to generate combinations or dispositions of (N,K) (all, random or a subset validated by an external application reading from standard input a partial or complete sequence), e.g.:

input

 @sequence 5,4
 ...
 @sequence 3^2
 ...
 @sequence 6,4 /my/external/validator
 ...
 @random 7 4
 ...
 @foreach 4,3
 the items of sequence are @1@, @2@ and @3@ (sequence @0@)
 @end

output

 1 2 3 4
 1 2 3 5
 1 2 4 5
 1 3 4 5
 2 3 4 5
 ...
 1,1
 1,2
 1,3
 2,1
 2,2
 2,3
 3,1
 3,2
 3,3
 ...
 1 4 5 6
 2 3 4 5
 ...
 3
 2
 7
 1
 ...
 the items of sequence are 1, 2 and 3 (sequence 1 2 3)
 the items of sequence are 1, 2 and 4 (sequence 1 2 4)
 the items of sequence are 1, 3 and 4 (sequence 1 3 4)
 the items of sequence are 2, 3 and 4 (sequence 2 3 4)

and other features such as creating a readable archive of a text files…

For details on MySTE see MySTE

MySDE

MySDE (My Service Development Environment) is a lightweight UNIX development environment makefile based for C/C++ and Java software projects.

It is the development environment of MySVC framework and includes support:

to build external libraries and binaries from packages sources handling multiple configurations for different UNIX environments
for configuration management (RCS)
for third party C/C++ and Java libraries (including J2EE web applications .war)
for Lex & Jacc compilers
for AsciiDoc based web sites
for FTP deploy

This environment is configured as:

a base environment (under $MYSDE_BASE)
a root environment, which is a subset of the base environment (under $MYSDE_ROOT=$MYSDE_BASE/src/<mysystem>)

The base environment contains:

all package sources and binaries
all makefiles to build binaries from sources
the support to create initial configuration for more UNIX environments and switch from one environment to another (each environment using its environment variable name for shared libraries)
the support to archive system binaries and sources

The root environment contains all system sources and its configured as a tree with three logical levels:

The system is the root of the tree
The subsystems are the childs of the system
The software units are the childs of each subsystem and the leafs of the tree

system
  |
  | ---- subsystem1
  |          |
  |          | ---- softwareunit11
  |          | ---- softwareunit12
  |          | ---- ...
  |          | ---- softwareunit1M
  | ---- subsystem2
  |          |
  |          | ---- softwareunit21
  |          | ---- softwareunit22
  |          | ---- ...
  |          | ---- softwareunit2M
  | ...
  | ---- subsystemN
             |
             | ---- softwareunitN1
             | ---- softwareunitN2
             | ---- ...
             | ---- softwareunitNM

Each of the three levels (system, subsystem and software unit) has an src subdirectory containing the corresponding Makefile to customize filling some variables

There is also a forth Makefile, included by other three makefiles, that is the "engine" of the development environment

The src subdirectories of the software units contain all the software (including scripts and configuration files)

Each software unit can generate one the following objects:

a C/C++ application
a C/C++ library (lib*.a or lib*.so) that can be used (linked):
- only from other software units of its own subsystem ("subsystem" library) or
- also from software units of other subsystems ("system" library)
a Java application
a Java library (.jar)
a Java/J2EE web application (.war)
an AsciiDoc based Web Site (with automatic FTP deploy)

From the system level is then possible to generate Java libraries (.jar) or web applications (.war)

Examples

The MySDE installation kit is made only of the mysde shell script.

In order to use MySDE you must:

* create the development environment executing mysde (specifying
  as optional argument its base directory):
      mysde
  (if not base directory is specified then $HOME/mysde is used as default)

* to use the development environment execute:
      . $HOME/mysde/.profile
  (to set all environment variables including
   $MYSDE_BASE as the base directory $HOME/mysde and
   $MYSDE_ROOT as the root directory $HOME/mysde/src/mysde,
   or $MYSYSTEM_BASE and $MYSYSTEM_ROOT if, for example,
   $HOME/mysystem was specified as base directory)

* if necessary customize file $MYSDE_BASE/.mysderc with
  environment variables, such as external PATH or
  LD_LIBRARY_PATH or others environment variables
  (or $MYSYSTEM_BASE/.mysystemrc if, for example,
   $HOME/mysystem was specified as base directory)

* copy under $MYSDE_BASE/pkg the source packages .tar.gz
  to be used by development environment (e.g. rcs-5.7.tar.gz),
  buildable with commands such as:
  configure
  make
  make install

* customize $MYSDE_BASE/makefiles/Makefile to add actions
  to build each source package copying the predefined templates
  build.mypkg and clean.mypkg (or customizing it if the commands
  specified into build.mypkg.template and clean.mypkg.template
  templates are not ok to build or clean the package)
  build.rcs-5.7:
        make build package=rcs-5.7

  clean.rcs-5.7:
        make clean package=rcs-5.7

* build each package executing under $MYSDE_BASE:
  make build.rcs-5.7

* add some subsystem and/or softwareunit to development
  environment:
      mysde -S mysubsystem1/mysoftwareunit1,mysubsystem2/mysoftwareunit2
  (or mysde -S mysubsystem1/mysoftwareunit1,mysubsystem2/mysoftwareunit2 $HOME/mysystem
   if, for example, $HOME/mysystem was specified as base directory)

* another way to add a subsystem is to execute under $MYSDE_ROOT/src
      make init subsystems="mysubsystem"

* another way to add a software unit is to execute under $MYSDE_ROOT/src/mysubsystem/src
      make init softwareunits="mysoftwareunit"

* customize $MYSDE_ROOT/src/Makefile specifying into variable SUBSYSTEMS
  the list of added subsystems (using one backslash as line terminator,
  for each line, if muliple lines are used to specify the list
  instead of spaces)

* for each subsystem customize $MYSDE_ROOT/src/mysubsystem/src/Makefile
  specifying into variable SOFTWAREUNITS the list of its softwareunits

* for each softwareunit of each subsystem copy under
  $MYSDE_ROOT/src/mysubsytem/src/mysoftwareunit/src the source code and
  configuration files used by that softwareunit

* for each softwareunit of each subsystem customize
  $MYSYSTEM_ROOT/src/mysubsytem/src/mysoftwareunit/src/Makefile
  specifying values for variables corresponding to source code
  files, binary targets, library paths, library names  or other
  options (e.g. for a C++ main application TARGET_BIN for the name
  of executable and SOURCES.cc and SOURCES.hh the name of C++ source
  code files)

* build the softwareunit and install (e.g. under $MYSDE_ROOT/bin,
  $MYSDE_ROOT/lib, $MYSDE_ROOT/classes of $MYSDE_ROOT/web)
  its deliverables (e.g. C++ executable, Java or C++ library)
  executing under $MYSDE_ROOT/src/mysubsytem/src/mysoftwareunit/src:
      make
      make install

* to backup only the development environment root tree execute
  under $MYSDE_ROOT/src:
      make tar
  (creating under $MYSDE_ROOT/.. a file mysde-YYYYMMDDhhmmssr-src.tar.gz)

* to backup your entire development environment base tree execute
  under $MYSDE_BASE:
      make tar
  (creating under $MYSDE_BASE/.. a file mysde-YYYYMMDDhhmmss.tar.gz)

* to use the development environment under another UNIX OS
  (e.g. a sun) is sufficient
    - make a backup of the entire development environment
      base tree
    - copy via ftp this backup file mysde-YYYYMMDDhhmmss.tar.gz
      on the new sun host
    - untar the backup:
        gunzip mysde-YYYYMMDDhhmmss.tar.gz
        tar xvf mysde-YYYYMMDDhhmmss.tar
    - execute . $MYSDE_BASE/.profile to init all environment
      variables
    - setup a new environment executing under $MYSDE_BASE:
      make init names="sun"
      (to create a new default set of sun configuration files)
      make conf name="sun"
      (to switch all symbolic links
       to sun files $MYSDE_BASE/.profile.sun, $MYSDE_BASE/.mysderc.sun,
       $MYSDE_BASE/local.sun, $MYSDE_BASE/.build.sun and
       $MYSDE_BASE/makefiles/Makefile.sun)
    - customize (if necessary) $MYSDE_BASE/.mysderc and
      $MYSDE_BASE/makefiles/Makefile
    - rebuild all packages from their sources executing the same
      "make build" commands from $MYSDE_BASE
      (e.g. make build package=rcs-5.7)
    - rebuild all subsystems and sofwareunit, e.g. executing
      from $MYSDE_ROOT/src:
          make clean
          make

* to return developing on the previous UNIX OS (e.g. "default"
  created as default when setup the development environment)
  is sufficient:
    - backup the entire development environment base tree
    - copy and untar this backup on the previous UNIX
    - execute:
        make conf name="default"
        (to switch all symbolic links
         to default files $MYSDE_BASE/.profile.default, $MYSDE_BASE/.mysderc.default,
         $MYSDE_BASE/local.default, $MYSDE_BASE/.build.default and
         $MYSDE_BASE/makefiles/Makefile.default)
    - rebuild all subsystems and sofwareunits executing
      from $MYSDE_ROOT/src:
          make clean
          make

* to get help on make commands availables execute
  from $MYSDE_BASE and $MYSDE_ROOT/src:
      make help

* two main rules when edit a Makefile:
    - when assign a value to a variable
        don't put spaces after equal ('=')
        if a value is multi-line end each line
        with a backspace ('\')
    - each command of a target must begin
      with a single tab character
  use ": set list" editing files with vi to view
  the hidden characters such as tabs and end of lines
  (then ":set nolist" to switch off)
  look at $MYSDE_ROOT/src/makefiles/Makefile as example

For details on MySDE see MySDE

MySPL

MySPL (My Service Protocol Library) is a C++ library to implement any TCP/IP socket client/server for binary or ASCII protocols.

For example, to implement a socket client for a binary protocol you can use the library classes defining your protocol messages from their fields (each of its type and size):

...
//
// VERSION
//

class UPSPVersion: public Message
{
  public:
    UPSPVersion():
      Message(5,"VERSION")
    {
      // Header fields
      assign(0,new Int4Field("length",18));
      assign(1,new Int2Field("msg_type",UPSP_VERSION));
      assign(2,new Int4Field("client_data_1"));
      assign(3,new Int4Field("client_data_2"));

      // Body field
      assign(4,new Int4Field("version",4));
    }
};

//
// LOGIN_REQ
//

class UPSPLoginReq: public Message
{
  public:
    UPSPLoginReq():
      Message(6,"LOGIN_REQ")
    {
      // Header fields
      assign(0,new Int4Field("length",71));
      assign(1,new Int2Field("msg_type",UPSP_LOGIN_REQ));
      assign(2,new Int4Field("client_data_1"));
      assign(3,new Int4Field("client_data_2"));

      // Body fields
      assign(4,new StringField(32,"username"));  // to init
      assign(5,new StringField(25,"password"));  // to init
    }
};

//
// LOGIN_ACK
//

class UPSPLoginAck: public Message
{
  public:
    UPSPLoginAck():
      Message(4,"LOGIN_ACK")
    {
      // Header fields
      assign(0,new Int4Field("length",14));
      assign(1,new Int2Field("msg_type",UPSP_LOGIN_ACK));
      assign(2,new Int4Field("client_data_1"));
      assign(3,new Int4Field("client_data_2"));
    }
};

//
// LOGIN_NACK
//

class UPSPLoginNAck: public Message
{
  public:
    UPSPLoginNAck():
      Message(6,"LOGIN_NACK")
    {
      // Header fields
      assign(0,new Int4Field("length",530));
      assign(1,new Int2Field("msg_type",UPSP_LOGIN_NACK));
      assign(2,new Int4Field("client_data_1"));
      assign(3,new Int4Field("client_data_2"));

      // Body fields
      assign(4,new Int4Field("error_code"));
      assign(5,new StringField(512,"error_msg"));
    }
};
...

and then using them:

creating a SocketClient from ip address and port
instantiating the message objects
implementing the workflow corresponding to the protocol reading and writing messages simply using the operators << and >>, or accessing their fields by name or by index:

  ...
  try
  {
    SocketClient wildfire(ipaddress,port);

    // Connection to wildfire ...

    if(upsp_log)
      cerr << "Connecting to Wildfire ..." << endl;

    wildfire.connect();

    UPSPHeader               header;

    UPSPVersion              version;
    UPSPLoginReq             login_req;
    ...
    int msg_type;
    ...

    // Login ...

    if(upsp_log)
      cerr << "Sending version ..." << endl;

    version[4] = "1.0";

    if(upsp_log)
    {
      if(upsp_trace)
        cerr << version << endl;
      cerr << "  <version> = <"
           << ((Int4Field&) version[4]).value()
           << ">" << endl;
    }

    wildfire << version;

    login_req[4] = "myusername";
    login_req[5] = "mypassword";

    if(upsp_log)
    {
      cerr << "Sending login_req ..." << endl;
      if(upsp_trace)
        cerr << login_req;
      cerr << "  <username> = <" << username << ">" << endl;
      cerr << "  <password> = <" << password << ">" << endl;
    }

    wildfire << login_req;

    if(upsp_log)
      cerr << "Reading ..." << endl;

    wildfire >> header;

    if(upsp_log)
    {
      if(upsp_trace)
        cerr << header << endl;
    }

    msg_type = (Int2Field&) header[1];

    switch(msg_type)
    {
      case UPSP_LOGIN_ACK:
      {
        if(upsp_log)
          cerr << "Received login_ack ..." << endl;

        break;
      }
      case UPSP_LOGIN_NACK:
      {
        if(upsp_log)
        {
          cerr << "Received login_nack ..." << endl;
          cerr << "Reading login_nack body ..." << endl;
        }

        wildfire >> login_nack_body;

        if(upsp_log)
        {
          if(upsp_trace)
            cerr << login_nack_body << endl;
          cerr << "  <error_code> = <"
               << ((Int4Field&) login_nack_body[0]).value()
               << ">" << endl;
          cerr << "  <error_msg> = <"
               << ((StringField&) login_nack_body[1]).value()
               << ">" << endl;
        }

        break;
      }
      ...
   }
   ...
  }
  catch(SocketException e)
  {
    cerr << e << endl;

    return WF_CONNECTION_ERROR;
  }
...

With this library you can also implement a simple ASCII protocol or define tree messages to implement complex binary protocols

For details on MySPL see MySPL

MyExpr

MyExpr (My Service Expression Engine) is a Java library (myexpr.jar) implementing a general purpose parser for expressions based on precedence operator parsing, using regular expressions to define lexical analyzer and generating a binary syntax tree as parser output.

The parser convert the string representation of the expression to its corresponding binary syntax tree and from binary syntax tree (an instance of Expr class or subclass) you can obtain the corresponding string representation.

E.g.:

import it.mysvc.expr.*;
...
Parser parser = new Parser(new MyExprOperators(),new MyExprLexer(),new MyExprFactory());
...
Expr expr = parser.parse("x*(y+-z))");
...
...
Expr left = expr.getLeft();
Expr right = expr.getRight();
String info = expr.getInfo();
int code = expr.getCode();
...

You must only implement some interfaces, e.g. to define unary and binary operators:

...
public class MyExprOperators implements Operators
{
  // binary operators
  public static final int AND         = 0;
  public static final int OR          = 1;
  public static final int IMPLIES     = 2;
  public static final int EQUIVALENT  = 3;
  public static final int ASSIGN      = 4;

  // unary operator
  public static int NOT               = 0;

  // constants TRUE and FALSE
  public static final int TRUE        = -1;
  public static final int FALSE       = 0;

  public int getBinaryOperators()
  {
    return 5;
  }

  public int getPrecedence(int binaryOperator)
  {
    switch(binaryOperator)
    {
      // AND
      case 0:
        return 5;
      // OR
      case 1:
        return 4;
      // IMPLIES
      case 2:
        return 3;
      // EQUIVALENT
      case 3:
        return 2;
      // ASSIGN
      default:
        return 1;
    }
  }

  public boolean isLeftAssociative(int binaryOperator)
  {
    return true;
  }

  public boolean isPrefix(int unaryOperator)
  {
    return true;
  }

  public boolean isPostfix(int unaryOperator)
  {
    return true;
  }
}
...

or to define the regular expression for unary and binary operators, operands and parenthesis:

...
// MyExprLexer:

public String[] getBinaryOperators()
{
  return new String[]
  {
    "&|and|\\*",    // code == 0
    "\\||or|\\+",   // code == 1
    "->|implies",   // code == 2
    "<->|iff",      // code == 3
    "=|is"          // code == 4
  };
}
...
public String[] getUnaryOperators()
{
  return new String[]
  {
    "-|~|!|not",
    "\\[[0-9]*\\]",
    "<[0-9]*>"
  };
}
...

or to get expression string corresponding to operators and operands:

// MyExprWriter:
...
public String getBinaryOperator(int code,String info)
{
  switch(code)
  {
    case MyExprOperators.AND:
      return "&";
    case MyExprOperators.OR:
      return "|";
    case MyExprOperators.IMPLIES:
      return "->";
    case MyExprOperators.EQUIVALENT:
      return "<->";
    case MyExprOperators.ASSIGN:
      return "=";
   }

   return null;
}
...

For details on MyExpr see MyExpr

Resources