Author Archive

SyncRemoting cookbook

June 4th, 2008

I wrote up a cheat sheet for myself when building scatter-gather or map reduce apps with GigaSpacesXAP.

It goes like this:

Create business interface (as part of shared classes)

Space-side stuff:

Create the Space-side implementation (logic) [ This may use a GigaSpace reference ]
Configure the implementation on the Space-Side:
Declare the business implementation as a [bean]
Declare a filter as part of the space
The filter references a ServiceExporter
Declare the os-remoting:service-exporter
The exporter references the business implementation [the bean]
If impl uses space: Declare the giga-space-context

Client-side stuff

Declare os-remoting:sync-proxy with its attributes:
The business interface it implements
The giga-space it uses as a transport layer
Whether broadcast is true or false
If necessary –declare the reducer for the proxy
Inject the client with the service proxy (ie: public void setMyService(MyService ref){} )

Let’s say we want to do a Map reduce exercise where each node in the cluster will process its share in parallel and return a total result for my digestion…

Let’s say I want to know the average price for all my widgets in all my inventory, but my inventory is scattered across a dozen servers because I do so much business…

I have my widgets:

package com.test.common;

import com.gigaspaces.annotation.pojo.SpaceRouting;

public class Widget {

private Double price;

private String description;

private Integer routingValue;

public Double getPrice() {
  return price;
}
public void setPrice(Double price) {
  this.price = price;
}
public String getDescription() {
  return description;
}
public void setDescription(String description) {
  this.description = description;
}
@SpaceRouting
public Integer getRoutingValue() {
  return routingValue;
}
public void setRoutingValue(Integer routingValue) {
  this.routingValue = routingValue;
}
}

My Service needs an interface such as:

package com.test.common;
public interface PricingService{

public Double getAveragePriceOfAllWidgets();
}

Next, I need to implement the service:

package com.test;

import org.openspaces.core.GigaSpace;
import org.openspaces.core.context.GigaSpaceContext;
import com.j_spaces.core.client.SQLQuery;
import com.test.common.PricingService;
import com.test.common.Widget;

public class PricingServiceImpl implements PricingService{
@GigaSpaceContext
GigaSpace space;

public Double getAveragePriceOfAllWidgets() {
  Object[] allWidgets = space.readMultiple(new Widget(), Integer.MAX_VALUE);
  double priceTotal = 0.0d;
  for(int x=0;x<allWidgets.length;x++){
   priceTotal=priceTotal+((Widget)allWidgets[x]).getPrice().doubleValue();
  }
  double averagePrice = priceTotal/allWidgets.length;
  return new Double(averagePrice);
}
}

Now I need a client that uses this service:

package com.test;
import java.util.TimerTask;
import com.test.common.PricingService;

public class WidgetSurfingClient extends TimerTask{

private PricingService service;
public void setPricingService(PricingService ref){
this.service=ref;
}
public void run(){
System.out.println(
”The average price of all widgets is now: $”+
service.getAveragePriceOfAllWidgets());
}
}

Now I need something to do the “reduce” side of the map-reduce: (it also performs the final average calculation because I am looking for the average price…

package com.test;
import org.openspaces.remoting.RemoteResultReducer;
import org.openspaces.remoting.SpaceRemotingInvocation;
import org.openspaces.remoting.SpaceRemotingResult;

public class AveragingDoubleResultReducer implements RemoteResultReducer<Double,Double>{

public Double reduce(SpaceRemotingResult<Double>[] results,
SpaceRemotingInvocation remotingInvocation) throws Exception {
  double totalDouble = 0.0d;
  int size = results.length;
  for (int x=0;x<size;x++){
   if(results[x]!=null){
    if(results[x].getResult()!=null){
      totalDouble += (Double) results[x].getResult();
    }
   }
  }
  double averageDouble=totalDouble/size;
  return new Double(averageDouble);
}
}

That takes care of the Java code, now we have to configure this system:

First: the Space-side configuration:

<!– declare the transaction manager –>
<os-core:local-tx-manager id=”transactionManager”
space=”widgetTransport”/>

<bean id=”PricingService” class=”com.test.PricingServiceImpl”/>

<!– provide a gigaspace context to the pricing service –>
<os-core:giga-space-context/>

<!–
Provide the binding to the Space-side
transport for the PricingService :
–>
<os-remoting:service-exporter id=”PricingServiceExporter”>
<os-remoting:service ref=”PricingService”/>
</os-remoting:service-exporter>

<!– configure a big or small cluster
(here we have 20 partitions with 1 backup for each) –>
<os-sla:sla cluster-schema=”partitioned-sync2backup”
number-of-instances=”20″ number-of-backups=”1″
max-instances-per-vm=”1″>
</os-sla:sla>

</beans>

Next: the client-side configuration:

<!– bootstrap the basic proxy for the serviceCluster: –>
<os-core:space id=”widgetTransport” url=”jini://*/*/widgetsystem” />

<!– wrap the proxy in Spring-ready OpenSpaces API –>
<os-core:giga-space id=”gigaspace” space=”widgetTransport” />

<!– WidgetSurfer Service –>
<bean id=”WidgetSurfer” class=”com.test.WidgetSurfingClient”>
<property name=”pricingService” ref=”PricingService”/>
</bean>

<os-remoting:sync-proxy id=”PricingService”
            giga-space=”gigaspace”
            broadcast=”true”
            interface=”com.test.common.PricingService”>
<os-remoting:result-reducer ref=”AveragingDoubleResultReducer”/>
</os-remoting:sync-proxy>

<!– this is regular Spring TimerTask configuration stuff: –>
<bean id=”widgetClientTask” class=”org.springframework.scheduling.timer.ScheduledTimerTask” >
<!– wait 1 seconds before starting repeated execution –>
<property name=”delay” value=”1000″ />
<!– run every 5 seconds –>
<property name=”period” value=”5000″ />
<property name=”timerTask” ref=”WidgetSurfer” />
</bean>

<bean id=”widgetClientTimer” class=”org.springframework.scheduling.timer.TimerFactoryBean”>
   <property name=”scheduledTimerTasks”>
    <list>
      <!– This wires the factory to the scheduledTask bean above –>
      <ref bean=”widgetClientTask” />
    </list>
   </property>
</bean>
</beans>

Cheers,

Owen.

Amazon gets more elastic

March 27th, 2008

Amazon EC2 now makes it possible for failure to be less noticeable within EC2 due to the new ability offered to have an IPAddress move from box to box upon failure. This is really cool because it means you can host your webserver as a service within the GigaSpaces AMI and when it automatically fails over due to our service management and self-healing behaviors, the end user will have a hiccup but will be able to refresh and continue without changing the URL!

Another cool new thing EC2 is offering is the ability to host primary and backup nodes in different physical “Zones” - meaning Amazon will guarantee that the backup location is not on the same power grid as the primary. Using GigaSpaces to create and deploy the instances makes it even easier to ensure this desired behavior is achieved.

See Amazon’s Feature Guides for Elastic IP Addresses and Availability Zones.

Learn how to build your first EC2 Applications using GigaSpaces

Cheers,

Owen.

Why not use maven?

January 16th, 2008

I received an (anonymous) comment regarding the command-line project creator.

(anonymous)asks:

“

why dont you use Maven archetypes for this ? See http://maven.apache.org/guides/introduction/introduction-to-archetypes.html

“

I answer:

First of all, if you like Maven I have good news!

The official GigaSpacesXAP product will support Maven and supply some archetypes very soon. (I believe the next early-access build will contain some support for this)

Now, you may, like Charles Miller , not like Maven.

The command-line project-creator is designed to be extremely light-weight and simple for anyone to use.

To use it, you need the jar file ~80kb and a probably two or three batch or shell scripts.

[Once I write some decent documentation, it will be more obvious what is required to execute the functionality.]

While it does create eclipse files, it also creates a build.xml suitable for execution from any ide or even emacs.

That said, once the maven stuff is supported and available, I plan to contribute my service templates to the available archetypes where they can add value.

I do not really see these as competing, but rather co-existing, like items on a menu: some people prefer fish, some meat, both have to eat.

I hope this helps to explain the existence of CPC

BTW: from now on, I will try to have news regarding CPC posted at openspaces and hope further comments can also live there, just for ease of information access and organization.

Cheers,

Owen.

Command-line Project creator (CPC) now on openspaces.org

January 15th, 2008

I have done it. I committed the project creator - now with support for pre-existing and shared domains - to openspaces.org.

I created a new tutorial video which currently acts as the documentation.

That can be seen here

I will be updating docs and such over the next few days.

The user interface now looks like this:

Welcome to the GigaSpaces Project Creator To create a new project, please answer the following questions:


What would you like to name your project [Worker]? myProcessor


Root directory for project creation [C:\_D\_wrk\projectcreator\out]?


GigaSpaces root directory [c:\GigaSpacesXAP6.0]?


Is Space embedded in this PU? [yes]


Name of space [gigaspace]


Do you wish to create the common domain [yes]?


*


What is the package name for the common domain [com.test.common] or ‘no’? org.test


*


Name of Common Object []? Message


*


Package name for generated service [com.test]?


*


ClassName of Service [MyService]? Processor


Created directory C:\_D\_wrk\projectcreator\out\myProcessor


commonDomainFilePath= commonDomain1177f763b16\common\domain


projDir = C:\_D\_wrk\projectcreator\out\myProcessor


Created directory C:\_D\_wrk\projectcreator\out\myProcessor\..\commonDomain1177f763b16\common\domain


commonDir = C:\_D\_wrk\projectcreator\out\myProcessor\..\commonDomain1177f763b16\common\domain


domainPackageDir = org\test


Created directory C:\_D\_wrk\projectcreator\out\myProcessor\..\commonDomain1177f763b16\common\domain\org\test
        DomainCreationHandler: Debug C:\_D\_wrk\projectcreator\out\commonDomain1177f763b16\common\domain


Created file C:\_D\_wrk\projectcreator\out\myProcessor\..\commonDomain1177f763b16\common\domain\org\test\Message.ja


Created directory C:\_D\_wrk\projectcreator\out\myProcessor\src


Created directory C:\_D\_wrk\projectcreator\out\myProcessor\src\test


Created directory C:\_D\_wrk\projectcreator\out\myProcessor\src\java


Created directory C:\_D\_wrk\projectcreator\out\myProcessor\src\java\com\test


Created file C:\_D\_wrk\projectcreator\out\myProcessor\src\java\com\test\Processor.java


Created directory C:\_D\_wrk\projectcreator\out\myProcessor\pu\myProcessor\META-INF\spring


Created directory C:\_D\_wrk\projectcreator\out\myProcessor\pu\myProcessor\META-INF\spring\mode


Created directory C:\_D\_wrk\projectcreator\out\myProcessor\src\java\META-INF\spring


Created file C:\_D\_wrk\projectcreator\out\myProcessor\pu\myProcessor\META-INF\spring\pu.xml


Created file C:\_D\_wrk\projectcreator\out\myProcessor\src\java\META-INF\spring\pu.xml


Created file C:\_D\_wrk\projectcreator\out\myProcessor\build.xml


Created directory C:\_D\_wrk\projectcreator\out\myProcessor\.eclipse


Created file C:\_D\_wrk\projectcreator\out\myProcessor\.eclipse\myProcessor.launch


EclipseTemplatePath= templates/eclipse/


Created file C:\_D\_wrk\projectcreator\out\myProcessor\.project


Created file C:\_D\_wrk\projectcreator\out\myProcessor\.classpath
        THE END.


        (We have completed all processing possible…)


        Hit Enter to continue…

So head over to

openspaces.org

-and check out the new version! (There is a zip file in the downloads section.)

Cheers,

Owen.

Opinionated architecture - blue prints without the middleman

December 16th, 2007

While at the Spring Experience this past week, I was delighted to learn a few things.

I got a good look at an OSGI environment, discovered what it felt like to turn rapidly in a circle on a cigarette boat doing well over 40 knots, and was introduced to the term “Opinionated Architecture”.

To my new understanding, an opinionated architecture, or framework, or solution, is one that proclaims a mandatory set of conventions in various things such as directory structure, configuration and code artifacts, naming conventions, and patterns of use. Due to the specification of these and possibly other things, the adopter of the solution will benefit from the use of the naturally occurring wizards and reusable templates that exemplify the mandatory bits and pieces. More than the traditional “blue-prints” offered by some, the essence of the available and proper use of the

opinionated solution depends on the erasure of alternative paths lest disaster strike.

This is not a trivial course to take. The author of the solution must have effective and real knowledge of the problem domain addressed by the solution - hopefully informed by enough failures that the successful path as declared is completely valid to the limits of the conceit of the solution.

For instance, web frameworks such as Grails provide rapid application development to those who agree to adopt the programming paradigm made possible and pushed by the elected framework. The result is a religious fervor adopted as those who share the faith and bend to the will of the higher, more-knowing, technological power are validated by their successes and spurred to greater adoption and recognition of the awesome power of the solutions’ prescription.

Caution: some wordiness follows:

The topic of the effect of poor planning and proceeding with lack of effective guidance down the enterprise architectural path was touched on in a BOF on high-performance Spring apps that took place late Friday night. (After the keynote filled with Monty Python references) During this talk, it was made clear that only experience can fulfill the needs of the architect challenged by a huge problem. What was also revealed is that humans frequently fail to avoid making the same mistakes numerous times even when presented with compelling alternatives. Old habits die hard it seems.

I am (tonight) of the opinion that unless your solution infrastructure mandates a path of best results and guides you to a proven successful state, you are more than likely to wander into the woods of competing best practices overgrown with budget constraints, lack of innovative and experienced resources, and just plain lack of understanding. It is therefore better to adopt a winning solution set whole-heartedly and when all the Kool-Aid is gone, agree to be dictated to by a benevolent dictator who has the right stuff.

The experience of a journeyman or artist is really very similar. In classical Ballet, for instance, the student rarely questions the validity of the techniques demonstrated by the Maestro leading the class/company. There is an element of necessary blind faith in the chosen path and the guides you choose to follow down it. Once one determines that the task at hand is indeed Ballet, one really has no choice but to turn out hips, point feet, and develop habits of posture that will sustain them through the most grueling choreography - for it will surely come and in spades. The student/apprentice is not well served to strike out on their own until a mastery is attained of the subject matter *through experience* that validates new choices and proves that through their adoption, greater success is to be found. The opinion therefore exhibited must be one based on hard-earned experience and still based predominately on a foundation made real and sustainable by the experiences of many others.

As it happens, I have watched the growth of OpenSpaces over the last many months with the skeptical eye of a - er skeptic - not realizing that the purpose of the framework is not only buzzword compliance and the implementation of popular development practices, but in the context of GigaSpaces and what is offered to our prospects and customers, a truly opinionated framework that takes the vastly flexible GigaSpaces infrastructure and service implementations and distills *the* successful use of it all into enforceable use of the Spring programmatic approach and technologies, thus empowering the neophyte and thought-leader alike with the unmistakably opinionated path to success.

The beauty of the switch to the prescriptive and formulaic from the free and often flailing style, is the stunningly rapid improvement in adoption rate, early prototyping successes, and sustained production-systems that continue to reinforce my growing certainty that we got our bit of it right this time.

Yes, OpenSpaces and by our support of it, GigaSpaces, has become an opinionated architecture/solution. As more and more tools and samples become available, and more wizards like my somewhat primitive, yet highly effective Project-creator, are built and offered, that fact will be obvious to all and joyfully so to the initiates and practitioners of the Linearly Scaling Aplication Arts. (The LSAA)

Write Once, [our way]

Scale Anywhere

Cheers,

Owen.

Update on Virgin Mobile post (thanks to Julian Browne)

December 13th, 2007

A while ago I posted a note about the success Virgin mobile experienced from switching to a Space-Based Architecture.

I received a comment from Julian Browne that I think is informative and so I make it public here:

***

Hi Owen - a backend system was unavailable for a few hours (honestly I

can’t remember the exact time). It was towards the end of the day’s

activity and it wasn’t back online until after midnight.

We knew this because one of the business managers thought they spotted

a bug in our reporting software after it informed him that more than

100% of orders had just been fulfilled. Never having seen anything

more than a 100% fulfilment rate, he naturally suspected an error.

On investigation we discovered that some of the previous day’s late

orders had been held in the space until the legacy system came back up

and submitted to the warehouse that morning. The system that was

replaced by the space-based approach didn’t do this and would often

lose orders entirely. What he really loved about this was that

managing orders safely in the event of a backend system not being

ready hadn’t even on his list of requirements.

I think there’s been some confusion created by other posts about the

project around order numbers generally. The number of online orders

increased by around three times almost immediately we went live (IT

wasn’t aware of this for a while, the system just scaled up without

any complaint) and over the Christmas period (a stressful time under

the pre-SBA solution) we handled a massive increase in orders

completed (unfortunately I’m not at liberty to say by how much because

it’s commercially sensitive).

I spoke to the online director only a couple of weeks ago and he still

loves it nearly two years later.

***

Thanks Julian,

It is great to know that what was once only a few lines of test code has survived and blossomed into a reference implementation.

Cheers,

Owen.

OpenSpaces.org is alive — project creator will move there soon

December 11th, 2007

GigaSpaces has started work on OpenSpaces.org and associated with the pending launch of the site is a call to arms for all who hack.

Check out: Open Spaces Developer Challenge I think there is even a sizable prize involved!

As soon as I can, I will make project creator available on OpenSpaces.org so anyone who wants to can improve it -extend it etc…

Cheers,

Owen.

My JayView Article is published!

November 8th, 2007

I am so excited that my article for JayView magazine. Has been published.

The title is:

Entering The Grid Painlessly - with spaces

The link for download is here:

http://www.jayway.se/jayview

You can also get “gasp” a hardcopy!

Cheers,

Owen.

Reminder of BOF session tonight

November 8th, 2007

I will be leading a Birds Of a Feather session at Qcon tonight at 7:15 in the City room of the Westin hotel in San Francisco.

Attendees will build a simple fault-tolerant, linearly scalable application using OpenSpaces and the GigaSpacesXAP platform.

Attendees will use the projectcreator to assist them in rapidly doing the above work.

I expect you will use either Eclipse version 3.3 europa or intellij as the development env, but as long as you have ant you should be good to go.

You can get the projectcreator.jar here:

http://www.jroller.com/resources/o/owentaylor/project-creator.jar

If you want to see it in use, you can look at the screencast I made a while back referenced here:

http://www.artvue-bermuda.com/scratchpad/projcreat1/ProjectCreator_publicDemo1.html

The files used in the video are available here:

http://www.jroller.com/resources/o/owentaylor/demoProjectCreator.jar

See you there!

Owen.

Freedom to Work and Play

November 8th, 2007

Holy good news Batman!

GigaSpaces is now free to use for startups!!!!

No. Really.

This is exciting because you now can get a full XAP license with all the bells and whistles to use forever. (as long as you do not earn more than 5 million dollars)

Everybody that wanted to try this stuff out but was concerned about being chased down by salespeople - have no fear. You can now use it in production for free as long as you are what GigaSpaces considers a startup.

GigaSpaces continues to offer the community edition for free to anyone at all.

Check out the announcement: here

Cheers,

Owen.

Well, what do you know?

November 4th, 2007

“Always make decisions based on what you know…”

A few years ago I received this excellent advice from a very successful businesswoman I worked for and I have tried to always follow it.

When I drive in a car in Manhattan I always look at the traffic on the street I am on before looking to switch to another avenue. If I “know” that 6th ave is moving at an average of 1 block a minute, unless I receive information that 8th ave or 10th ave is moving faster, or I “know” from prior experience that this is the case, I remain where I am and get to my destination in a reasonable amount of time.

The question remains: “How do we come to know things?”

It is all very well to make decisions based on what you know if what you know is up to date and based on the very best possible information. The problem is we are creatures of habit. We also like to attach meaning to memory and familiarity that has little or no basis in fact. For instance, I, despite my prior ramblings to the contrary have a belief that taking the west-side highway is faster than any of the west-side avenues because it is a highway. My experience when taking this highway shows me that in fact, the highway is often slower than say, 10th avenue, and by taking it I am impeding my progress in getting myself and any passengers to the destination intended. I am however, often found, moving at a snail’s pace on my precious highway, angry passengers beside me, cursing my bad luck.

–hint: ‘luck’ is *not* knowledge based–

I see the same kind of behavior exhibited in technologists who, once they get a certain idea about a technology, stick to that impression despite evidence to the contrary.

Sometimes, there are practical reasons for the adherence to a particular solution, such as the fact that they only last year invested 3 million dollars in the technology they cling to and to change their attitude or exposed belief would mean admission of a grave mistake and possibly the loss of their position in the company.

Sometimes, there are historical reasons for the solution-hugging, such as prior success and familiarity that means starting something new with that technology requires almost no learning aside from the business domain.

Sometimes, there are social pressures that cause an individual to pivot away from new intriguing solutions and side with what a particular leader or sensible seeming majority prefers.

In my opinion, only recent experience with a technology that shows success in achieving meaningful criteria, provides the knowledge vital to include such technnology in your basket of options.

As a proud employee of GigaSpaces, I have the pleasure to rattle the mental cages of many people throughout the world with what appear to be radical concepts and (potentially at least) highly disruptive technology - I am now referring to the Space-Based Architecture and linearly scalable infrastructure offered by the use of our middleware. When people are presented with the facts surrounding the use of the GigaSpaces solution, they see with their own eyes the improved throughput and continuous availability and resiliency, and they get excited. For many, what they “know” has just changed. To these people, I offer my best advice,

“Always make decisions based on what you know.”

To those who to date have not had the successful knowledge transfer and remain with their tried and true and socially acceptable solutions, I say, “Risk, Reward.” Get educated in as many of the available solutions as possible. Step out of your comfortable shell long enough to learn what is possible in the innovative world of the data-grid and allow the Kool-Aid to take effect.

Kick the tires of this new sports car and see if the triple wammy of:

1) reliable and super-fast state management

2) dynamic service provisioning and recoverability

3) collocation of events, processing and information into a Spring-based, Java, .NET, and C++ friendly processing unit

- doesn’t offer you a quicker way to your technical destination than that well understood and familiar yet often disappointing “other” you cling to with such remorse.

You can always be last to go with the better technology, but don’t expect much success if you make it a habit. Curiously, one of the perhaps loose yet highly relevant definitions of insanity is repeating the same behavior over and over and expecting different results.

Cheers,

Owen.

P.S. To learn more about GigaSpaces and perhaps be taken out of your comfort zone in the process check out this page filled with links to great info on what is possible and how. You can also see me and the GigaSpaces CTO Nati Shalom, presenting at QCon next week in San Francisco.