SpatialKey 2011 – Year in Review

Lots of great things happening all the time at SpatialKey.  It has been an incredible year for us.  Check out this blog post detailing the many accomplishments:

http://blog.spatialkey.com/2012/01/what-we-did-in-2011-year-in-review/

SpatialKey, Salesforce and Geopointe

SpatialKey now offers its spatial analytics capabilities to Salesforce users by way of our partnership with Arrowpointe software’s Geopointe mapping solution for Force.com.  Geopointe is the #1 Force.com geo/mapping solution and by way of our partnership now adding full-featured Geo-Analytics as well.

If you currently use Salesforce and are looking for a mapping and analytics solution check out Geopointe!  Here is a link to the Geopointe blog talking about the new SpatialKey integration, oh and a video to!

http://www.arrowpointe.com/2011/06/17/geo-analytics-in-salesforce/

 

 

 

 

 

 

 

 

 

Our Application Server Stack

With a new year coming up I thought I would give a quick run down of our OS server stack.  From the beginning SpatialKey has been built on a number of great community/company open source platforms, here is what we currently use:

1. Apache/Tomcat
1. We are currently evaluating moving our application servers to pure Tomcat vs. JBoss as at this time we are not taking advantage of any of the additional server capabilities JBoss provides and Tomcat gives a smaller and more easily managed servlet container.
2. JBoss AS
1. Has been our application server platform from the beginning.  I cannot say enough good things about JBoss.  It is an outstanding full stack J2EE server.
3. Postgres SQL 9 + PostGIS
1. SpatialKey would not be what it is today without Postgres and PostGIS.  Postgres is a great enterprise ready database from the get-go, but add PostGIS to it and you have a powerful full stack GIS database on top of it.
4. Blaze DS
1. The SpatialKey client is flash under the hood and Flex developed.  Blaze DS is the glue that allows the flash client to request server services.
5. Server Development
1. There are really too many technologies to list here, but there are three to definitely point out:
1. Spring
1. Ah, Spring… the magic container that brings the whole SpatialKey server together.  Don’t leave home without it!
2. Hibernate
1. Going hand in hand with Spring, Hibernate gives us an industry standard object relational mapping engine to help with the heavy lifting of standard database interactions.  Everything non-customer data goes through Hibernate in SpatialKey.
3. Railo
1. I have always been a ColdFusion fan.  I have worked on ColdFusion since its very early days and to this day believe it is still the easiest web application server to develop for.  Railo brings ColdFusion development to the open source world and of course at no charge!

So, there you have it the open source server stack used in SpatialKey (did I mention all of this great server goodness is free??).  If anyone has any specific questions regarding our usage of these products, please ask them in the comments and I will be glad to answer any that I can.

SpatialKey nearing 2.3

Wow, the past year has been one of excitement and growth for SpatialKey as a platform!  We are now nearing a 2.3 version release of the server.  Over the year we have made great inroads into the insurance industry and are focusing in on a new “Apps” based approach for adding specific targeted capabilities to SpatialKey.

The first of these Apps will be a Natural Perils suite that we have recently introduced.  Check out what we are working on here:

http://spatialkey.com/industry-solutions/insurance/details/instantly-forecast-hurricane-exposures/

Using SpatialKey to analyze how far closing Saturn dealerships are from other GM dealers.

A lot of news coming from SpatialKey over the last week.

First of all, SpatialKey is now officially out of beta and is live!  Free trial accounts are still available so check it out (completely re-designed web site too) – www.spatialkey.com!

Second, with news coming out yesterday that GM is going to be closing Saturn dealerships owners are concerned (and rightfully so) about where they will be able to get warranty service going forward.  To help answer the question of “How far are Saturn dealerships from another GM dealership?” is one that can be answered very easily by SpatialKey.  After doing a spatial analysis of data provided by AggData of Saturn and other GM dealerships the answer is a good one for most owners: there is at least one other GM dealership within 10 miles of all closing Saturn dealers.

Check out this blog post (including screenshots and an interactive SpatialKey report) showing this information here.

SpatialKey Nearing 1.0 Release!

These are exciting times for the SpatialKey team as we approach our 1.0 release!  Open beta has been going for a while now (you have signed up for a beta account haven’t you???) and the feedback we are getting from our users has been phenomenal!!  We have listened to what our users are telling us and continue to try and improve the product before launch.  So, thank you to everyone that has provided feedback, we are listening!

As I mentioned in the last SpatialKey update, part of the 1.0 release is a major UI enhancement.  We believe that the new report UI will make things easier to understand, become more intuitive, and expose features that you may have not even known were there.  We are very proud of it!

To illustrate some of these features we have a couple of new videos we are working on (still a work in progress) so check it out!

The first video (available via YouTube here) shows off a bunch of the new UI features.  It goes through a real world scenario of how one user may use SpatialKey to create a report.

The second is a shorter video that specifically targets some of the features surrounding our new “Proximity Filtering” capabilities (see it here).

It is still not too late to check out the Beta including these new features and UI before we go live.  To sign up go here!

SpatialKey News

Some exciting things are happening in the SpatialKey world.  We are nearing the end of the Beta cycle and will soon be in open Beta.  So, if you have not yet applied or been accepted into the current closed Beta you should get your chance soon.  To apply go here: www.spatialkey.com.  Once the open Beta begins it will just be a short time until we launch the 1.0 release.

The 1.0 release is going to include a report interface re-design that we are all very excited about.  Our team designer, Francisco Inchauste, is amazing.  Take a look at his blog post here for a tiny sample of the new look and feel.  Can’t wait to get it out!

Dynamic Scheduling with Java/Spring

SpatialKey owes much of its success on the server to the simplicity and capabilities of the Spring framework.  Spring provides many services that we take advantage of and one of these is the scheduling service.

The Spring scheduling service allows for the execution of tasks on a well… scheduled basis.  With basic Spring scheduling support a developer can use XML based bean wiring to run tasks either using CRON expressions or with a simple timer.  This is probably all most applications would ever need and is pretty easy to set up.

Typically, to run a scheduled task with Spring you need three things.

1. A SchedulerFactoryBean instance
2. One or more triggers (CRON or timer based)
3. One or more jobs

Here is an example of what this could look like in a Spring configuration file:

<!-- Start the Triggers -->
<bean id="scheduleFactory" class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
    <property name="triggers">
        <list>
            <ref bean="cronDailyFileCleanupTrigger"/>
        </list>
    </property>
</bean>

<!-- Cron Triggers -->
<bean id="cronDailyFileCleanupTrigger"
    class="org.springframework.scheduling.quartz.CronTriggerBean">
    <property name="jobDetail" ref="dailyFileCleanupJob"/>
    <property name="cronExpression" value="0 0 3 * * ?"/>
</bean>

<!-- Begin Jobs -->
<bean id="dailyFileCleanupJob"
    class="org.springframework.scheduling.quartz.JobDetailBean">
    <property name="jobClass" value="com.spatialkey.scheduledtasks.DailyFileCleanupJob"/>
    <property name="jobDataAsMap">
       <map>
           <entry key="systemProps" value-ref="systemProperties"/>
       </map>
    </property>
</bean>

The above code does the following:

1. The first block instantiates a SchedulerFactoryBean.  This is the class that manages the creation of schedules in the Spring framework.  This instance can have one or more “triggers”.  These triggers represent a single scheduled task and are defined in another bean (shown below).  The list can contain any number of triggers as needed.
2. The second block defines a trigger.  This particular trigger is a CRON based trigger (specified by the class CronTriggerBean) and requires a CRON expression telling the trigger when it should execute.  I am not going to go into the ins and outs of CRON expressions in this article, but this particular trigger is set up for 3 am.  The other property (jobDetail) is a reference to a bean instance defining the task to run.
3. The third block defines the actual job/class to run.  The class itself is defined by the programmer containing the code to execute on the scheduled basis.  Also notice that other beans can be injected (just like any other Spring beans) using the jobDataAsMap property (there are other ways as well).

Now, all of this is nothing new to Spring and this information is readily available (I also highly recommend the book Spring in Action by Craig Walls).  What I really wanted to talk about in this blog is how to dynamically create triggers and scheduled tasks programmatically as there is not a whole lot of information out there on how to do this.

The Dynamic Schedule Process

If you are wanting to dynamically create scheduled tasks you cannot rely on XML configuration.  Instead we will use a programmatic process to do a similar thing that happens with the configuration example above.  The general process is:

1. Wire up a SchedulerFactoryBean as above.  We will use this bean wired Spring service to create our schedules with.
2. In your code either inject the scheduler factory or access it using the Spring application context.
3. Get access to an instance of the object that will be executed on the timed basis.
4. Create an instance of a MethodInvokingJobDetailFactoryBean (thats a mouthful).  This object is a wrapper for a job object (that will get linked to a trigger) for execution.
5. Create an instance of a SimpleTriggerBean (for simple interval triggers) or a CronTriggerBean (for CRON based triggers).
6. Schedule the job with the scheduler.

Ok, so what does this look like in code?  Here is an example:

//get the quartzFactory bean
Scheduler scheduler = (Scheduler) ctx.getBean("scheduleFactory");

//get the task to run or it could have been injected
DataPollingTask dpTask = (DataPollingTask) ctx.getBean(taskName);

//this example uses a simple interval schedule, but could be done with a CRON schedule by using the correct Trigger Bean (CronTriggerBean)
//create job
jobDetail = new MethodInvokingJobDetailFactoryBean();
jobDetail.setTargetObject(dpTask);
jobDetail.setTargetMethod("run");
jobDetail.setName(taskName);
jobDetail.setConcurrent(false);
jobDetail.afterPropertiesSet();

//create trigger
SimpleTriggerBean trigger = new SimpleTriggerBean();
trigger.setBeanName(taskName);
trigger.setJobDetail((JobDetail) jobDetail.getObject());
trigger.setRepeatInterval(interval);
trigger.afterPropertiesSet();

//add to schedule
scheduler.scheduleJob((JobDetail) jobDetail.getObject(), trigger);

A couple of things to note here.  Notice the setTargetMethod call on the job detail is set to “run”?  This is telling the job when it executes what the method name is that should be executed.  This method must exist in the target object to be executed.  The method should be public and return nothing (void).  Also the method (in this case “run”) should not take any parameters.

Of course there is more than one way to do this whole process and there are other properties/methods/techniques available in the Spring scheduling engine so please take a look at the various JavaDocs for the objects listed for more information but with the code an process outlined above you should be well on your way to creating dynamic Spring based scheduled tasks.

Running a command line executable in Java with redirected output???

First off there are some awesome new features coming very soon to SpatialKey!  I can’t let the cat completely out of the bag, but by reading this article you may get some idea of what is coming…

One of the requirements for SpatialKey that has come up is the necessity to execute a command line tool from inside of our Java server implementation.  If you are at all familiar with SpatialKey (and if not, what are you waiting for??? get on over to www.spatialkey.com) you know that it is a geo-spatial application.  As such, we need to store and retrieve geo-spatial data and to accomplish this goal we have settled on using PostGreSQL along with PostGIS extensions.

One of the tools that comes with PostGIS is called shp2pgsql (shape to PostGreSQL).  This tool allows for taking a geo-spatial shape file and converting the contents into PostGreSQL compatable SQL for database insertion (hint… hint…).  These shapes can then be used for filtering in PostGIS SQL calls.

Here is a simple example of what using this tool would look like if run from a terminal window:

./shp2pgsql roads.shp roads_table my_db > roads.sql

Notice the output redirection ‘> roads.sql’ this unix operator simply dumps the result into the file specified and works great from a terminal window.  The generated sql could then be run against the database to import the shape data.  Unfortunately, however, this is not the case when run from within Java.

So, how do you run a command line tool from Java?  It is pretty easy actually.  And the good folks at Apache Commons have made the process pretty robust.  I recommend using their command line execution package (download here).

Using their code the above example might look something like this:

CommandLine cmdLine = CommandLine.parse("./shp2pgsql roads.shp roads_table my_db > roads.sql");
DefaultExecutor executor = new DefaultExecutor();
executor.execute(cmdLine);

Unfortunately this does not work.  The JVM thinks this entire line is a part of the shp2pgsql command when in reality they are two separate commands that the operating system needs to interpret.  If we leave off the “> roads.sql” then everything works as it should, but you have no captured SQL.  So how do we make this work?  We need to capture the output stream from the command line execution:

ByteArrayOutputStream bos = new ByteArrayOutputStream();
CommandLine cmdLine = CommandLine.parse("./shp2pgsql roads.shp roads_table my_db");
PumpStreamHandler psh = new PumpStreamHandler(bos, System.out);
DefaultExecutor executor = new DefaultExecutor();
executor.setStreamHandler(psh);
executor.execute(cmdLine);
bos.close();

So the code has changed a bit here.  First of all we define a ByteArrayOutputStream that will capture the data from the command execution.  This output stream is passed as an argument to a new instance of a PumpStreamHandler object (this object is a part of the Apache Commons Exec code base).  With this stream handler we can now tell the executor that we do not want the output stream to go to its default but instead to the ByteArrayOutputStream we defined.  Also notice that I removed the offending “> roads.sql” from the original command line.

Once I had access to the stream I simply ran a toString() on the contents and executed the SQL.  Of course if I really wanted the contents placed into a file I could have used a FileOutputStream instead.

That’s all there is to it.

Java Date Parsing (With an Unknown Format)

One of the biggest challenges within SpatialKey is how we could handle the many different date formats that a user could come up with.  And there are so many combinations of numbers (one digit month/day, two digit (month/day), two and four diget year, etc.) , words (Mon, Tue, Monday, Tuesday, Jun, Jul, June, July, etc.), and order of the parts that it is quite a daunting task.

The first solution was to create an acceptable list of formats and only parse for those.  We quickly realized that this (albeit an easy approach technically) would put a large data transformation headache on nearly all customers.  We really wanted a solution that would account for nearly all situations (although admitting that we could not support EVERY format imaginable).  The task then fell to me to figure out if/how we could make this a reality.

After looking all over for a good native Java way of doing this and coming up with nothing (pretty much everything needed a format string) I stumbled upon several forum posts referring to the POJava project.  This open source project contains many simple “Plain Old Java” object utilities but what most interested me was the date parser it comes with.  The POJava date parser org.pojava.datetime.DateTime is an immutable, and robust parser that supports multiple languages, time zones, and formats.  Best of all, the parser is heuristic and does not require a pre-existing “format” to work.  You just pass it a date/date-time text string and get out a java.util.Date!  It works great.  Will it do every possible format without exception, nope, but it will sure do the vast majority of them.

The only real issue I ran into was that the parser kept trying to put everything into the Pacific time zone and I wanted the dates to get stored in the server’s time zone.  To fix that I simple appended the appropriate time zone string to the end of the passed in date.  Easy.

So, if you have a similar requirement I highly recommend taking a look at POJava.