Using SpatialKey to analyze how far closing Saturn dealerships are from other GM dealers.

A lot of news coming from SpatialKey over the last week.

First of all, SpatialKey is now officially out of beta and is live!  Free trial accounts are still available so check it out (completely re-designed web site too) – www.spatialkey.com!

Second, with news coming out yesterday that GM is going to be closing Saturn dealerships owners are concerned (and rightfully so) about where they will be able to get warranty service going forward.  To help answer the question of “How far are Saturn dealerships from another GM dealership?” is one that can be answered very easily by SpatialKey.  After doing a spatial analysis of data provided by AggData of Saturn and other GM dealerships the answer is a good one for most owners: there is at least one other GM dealership within 10 miles of all closing Saturn dealers.

Check out this blog post (including screenshots and an interactive SpatialKey report) showing this information here.

SpatialKey Nearing 1.0 Release!

These are exciting times for the SpatialKey team as we approach our 1.0 release!  Open beta has been going for a while now (you have signed up for a beta account haven’t you???) and the feedback we are getting from our users has been phenomenal!!  We have listened to what our users are telling us and continue to try and improve the product before launch.  So, thank you to everyone that has provided feedback, we are listening!

As I mentioned in the last SpatialKey update, part of the 1.0 release is a major UI enhancement.  We believe that the new report UI will make things easier to understand, become more intuitive, and expose features that you may have not even known were there.  We are very proud of it!

To illustrate some of these features we have a couple of new videos we are working on (still a work in progress) so check it out!

The first video (available via YouTube here) shows off a bunch of the new UI features.  It goes through a real world scenario of how one user may use SpatialKey to create a report.

The second is a shorter video that specifically targets some of the features surrounding our new “Proximity Filtering” capabilities (see it here).

It is still not too late to check out the Beta including these new features and UI before we go live.  To sign up go here!

SpatialKey News

Some exciting things are happening in the SpatialKey world.  We are nearing the end of the Beta cycle and will soon be in open Beta.  So, if you have not yet applied or been accepted into the current closed Beta you should get your chance soon.  To apply go here: www.spatialkey.com.  Once the open Beta begins it will just be a short time until we launch the 1.0 release.

The 1.0 release is going to include a report interface re-design that we are all very excited about.  Our team designer, Francisco Inchauste, is amazing.  Take a look at his blog post here for a tiny sample of the new look and feel.  Can’t wait to get it out!

Dynamic Scheduling with Java/Spring

SpatialKey owes much of its success on the server to the simplicity and capabilities of the Spring framework.  Spring provides many services that we take advantage of and one of these is the scheduling service.

The Spring scheduling service allows for the execution of tasks on a well… scheduled basis.  With basic Spring scheduling support a developer can use XML based bean wiring to run tasks either using CRON expressions or with a simple timer.  This is probably all most applications would ever need and is pretty easy to set up.

Typically, to run a scheduled task with Spring you need three things.

  1. A SchedulerFactoryBean instance
  2. One or more triggers (CRON or timer based)
  3. One or more jobs

Here is an example of what this could look like in a Spring configuration file:

<!-- Start the Triggers -->
<bean id="scheduleFactory" class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
    <property name="triggers">
        <list>
            <ref bean="cronDailyFileCleanupTrigger"/>
        </list>
    </property>
</bean>

<!-- Cron Triggers -->
<bean id="cronDailyFileCleanupTrigger"
    class="org.springframework.scheduling.quartz.CronTriggerBean">
    <property name="jobDetail" ref="dailyFileCleanupJob"/>
    <property name="cronExpression" value="0 0 3 * * ?"/>
</bean>

<!-- Begin Jobs -->
<bean id="dailyFileCleanupJob"
    class="org.springframework.scheduling.quartz.JobDetailBean">
    <property name="jobClass" value="com.spatialkey.scheduledtasks.DailyFileCleanupJob"/>
    <property name="jobDataAsMap">
       <map>
           <entry key="systemProps" value-ref="systemProperties"/>
       </map>
    </property>
</bean>

The above code does the following:

  1. The first block instantiates a SchedulerFactoryBean.  This is the class that manages the creation of schedules in the Spring framework.  This instance can have one or more “triggers”.  These triggers represent a single scheduled task and are defined in another bean (shown below).  The list can contain any number of triggers as needed.
  2. The second block defines a trigger.  This particular trigger is a CRON based trigger (specified by the class CronTriggerBean) and requires a CRON expression telling the trigger when it should execute.  I am not going to go into the ins and outs of CRON expressions in this article, but this particular trigger is set up for 3 am.  The other property (jobDetail) is a reference to a bean instance defining the task to run.
  3. The third block defines the actual job/class to run.  The class itself is defined by the programmer containing the code to execute on the scheduled basis.  Also notice that other beans can be injected (just like any other Spring beans) using the jobDataAsMap property (there are other ways as well).

Now, all of this is nothing new to Spring and this information is readily available (I also highly recommend the book Spring in Action by Craig Walls).  What I really wanted to talk about in this blog is how to dynamically create triggers and scheduled tasks programmatically as there is not a whole lot of information out there on how to do this.

The Dynamic Schedule Process

If you are wanting to dynamically create scheduled tasks you cannot rely on XML configuration.  Instead we will use a programmatic process to do a similar thing that happens with the configuration example above.  The general process is:

  1. Wire up a SchedulerFactoryBean as above.  We will use this bean wired Spring service to create our schedules with.
  2. In your code either inject the scheduler factory or access it using the Spring application context.
  3. Get access to an instance of the object that will be executed on the timed basis.
  4. Create an instance of a MethodInvokingJobDetailFactoryBean (thats a mouthful).  This object is a wrapper for a job object (that will get linked to a trigger) for execution.
  5. Create an instance of a SimpleTriggerBean (for simple interval triggers) or a CronTriggerBean (for CRON based triggers).
  6. Schedule the job with the scheduler.

Ok, so what does this look like in code?  Here is an example:

//get the quartzFactory bean
Scheduler scheduler = (Scheduler) ctx.getBean("scheduleFactory");

//get the task to run or it could have been injected
DataPollingTask dpTask = (DataPollingTask) ctx.getBean(taskName);

//this example uses a simple interval schedule, but could be done with a CRON schedule by using the correct Trigger Bean (CronTriggerBean)
//create job
jobDetail = new MethodInvokingJobDetailFactoryBean();
jobDetail.setTargetObject(dpTask);
jobDetail.setTargetMethod("run");
jobDetail.setName(taskName);
jobDetail.setConcurrent(false);
jobDetail.afterPropertiesSet();

//create trigger
SimpleTriggerBean trigger = new SimpleTriggerBean();
trigger.setBeanName(taskName);
trigger.setJobDetail((JobDetail) jobDetail.getObject());
trigger.setRepeatInterval(interval);
trigger.afterPropertiesSet();

//add to schedule
scheduler.scheduleJob((JobDetail) jobDetail.getObject(), trigger);

A couple of things to note here.  Notice the setTargetMethod call on the job detail is set to “run”?  This is telling the job when it executes what the method name is that should be executed.  This method must exist in the target object to be executed.  The method should be public and return nothing (void).  Also the method (in this case “run”) should not take any parameters.

Of course there is more than one way to do this whole process and there are other properties/methods/techniques available in the Spring scheduling engine so please take a look at the various JavaDocs for the objects listed for more information but with the code an process outlined above you should be well on your way to creating dynamic Spring based scheduled tasks.

Running a command line executable in Java with redirected output???

First off there are some awesome new features coming very soon to SpatialKey!  I can’t let the cat completely out of the bag, but by reading this article you may get some idea of what is coming…

One of the requirements for SpatialKey that has come up is the necessity to execute a command line tool from inside of our Java server implementation.  If you are at all familiar with SpatialKey (and if not, what are you waiting for??? get on over to www.spatialkey.com) you know that it is a geo-spatial application.  As such, we need to store and retrieve geo-spatial data and to accomplish this goal we have settled on using PostGreSQL along with PostGIS extensions.

One of the tools that comes with PostGIS is called shp2pgsql (shape to PostGreSQL).  This tool allows for taking a geo-spatial shape file and converting the contents into PostGreSQL compatable SQL for database insertion (hint… hint…).  These shapes can then be used for filtering in PostGIS SQL calls.

Here is a simple example of what using this tool would look like if run from a terminal window:

./shp2pgsql roads.shp roads_table my_db > roads.sql

Notice the output redirection ‘> roads.sql’ this unix operator simply dumps the result into the file specified and works great from a terminal window.  The generated sql could then be run against the database to import the shape data.  Unfortunately, however, this is not the case when run from within Java.

So, how do you run a command line tool from Java?  It is pretty easy actually.  And the good folks at Apache Commons have made the process pretty robust.  I recommend using their command line execution package (download here).

Using their code the above example might look something like this:

CommandLine cmdLine = CommandLine.parse("./shp2pgsql roads.shp roads_table my_db > roads.sql");
DefaultExecutor executor = new DefaultExecutor();
executor.execute(cmdLine);

Unfortunately this does not work.  The JVM thinks this entire line is a part of the shp2pgsql command when in reality they are two separate commands that the operating system needs to interpret.  If we leave off the “> roads.sql” then everything works as it should, but you have no captured SQL.  So how do we make this work?  We need to capture the output stream from the command line execution:

ByteArrayOutputStream bos = new ByteArrayOutputStream();
CommandLine cmdLine = CommandLine.parse("./shp2pgsql roads.shp roads_table my_db");
PumpStreamHandler psh = new PumpStreamHandler(bos, System.out);
DefaultExecutor executor = new DefaultExecutor();
executor.setStreamHandler(psh);
executor.execute(cmdLine);
bos.close();

So the code has changed a bit here.  First of all we define a ByteArrayOutputStream that will capture the data from the command execution.  This output stream is passed as an argument to a new instance of a PumpStreamHandler object (this object is a part of the Apache Commons Exec code base).  With this stream handler we can now tell the executor that we do not want the output stream to go to its default but instead to the ByteArrayOutputStream we defined.  Also notice that I removed the offending “> roads.sql” from the original command line.

Once I had access to the stream I simply ran a toString() on the contents and executed the SQL.  Of course if I really wanted the contents placed into a file I could have used a FileOutputStream instead.

That’s all there is to it.

Java Date Parsing (With an Unknown Format)

One of the biggest challenges within SpatialKey is how we could handle the many different date formats that a user could come up with.  And there are so many combinations of numbers (one digit month/day, two digit (month/day), two and four diget year, etc.) , words (Mon, Tue, Monday, Tuesday, Jun, Jul, June, July, etc.), and order of the parts that it is quite a daunting task.

The first solution was to create an acceptable list of formats and only parse for those.  We quickly realized that this (albeit an easy approach technically) would put a large data transformation headache on nearly all customers.  We really wanted a solution that would account for nearly all situations (although admitting that we could not support EVERY format imaginable).  The task then fell to me to figure out if/how we could make this a reality.

After looking all over for a good native Java way of doing this and coming up with nothing (pretty much everything needed a format string) I stumbled upon several forum posts referring to the POJava project.  This open source project contains many simple “Plain Old Java” object utilities but what most interested me was the date parser it comes with.  The POJava date parser org.pojava.datetime.DateTime is an immutable, and robust parser that supports multiple languages, time zones, and formats.  Best of all, the parser is heuristic and does not require a pre-existing “format” to work.  You just pass it a date/date-time text string and get out a java.util.Date!  It works great.  Will it do every possible format without exception, nope, but it will sure do the vast majority of them.

The only real issue I ran into was that the parser kept trying to put everything into the Pacific time zone and I wanted the dates to get stored in the server’s time zone.  To fix that I simple appended the appropriate time zone string to the end of the passed in date.  Easy.

So, if you have a similar requirement I highly recommend taking a look at POJava.

Introducing the SpatialKey DataImportAPI!

As of today SpatialKey is introducing a much requested feature… a data import API!

We have received great feedback from the community during our beta process.  One request (amongst others) is that it would be great to have alternate ways of getting data into the SpatialKey environment without using the SpatialKey client.  You asked for it and we listened!  Enter the SpatialKey DataImportAPI.

The API is built around simple HTTP services that should allow developers the ability to dynamically push data to SpatialKey.  For more information on this new capability jump on over to the API documentation here.

We also have started an open source project at Google Code that includes a sample application written in Java called the SpatialKey Data Poller.  This application is designed to “read” from an FTP location and publish data to the API on a scheduled basis (it is also designed to have other polling types added – HTTP, directory, etc… if someone would like to add them please do and share).  The binary and code is available here.  In addition to the Java app, Brandon Purcell has created a Cold Fusion component (usable out of the box) that uses the API too (also available on the downloads page of the Google Code project) so there are some good examples available.

So have at it guys and let us know what you think of the documentation, code, and the API itself!

Oh, one other thing!  SpatialKey also has another product available called the Enterprise Data Publisher.  This product is designed to run behind a clients firewall and push data on a scheduled basis from a database to SpatialKey.  If you are interested in this product (and do not want to create something custom using the API) contact our sales team.

New SpatialKey Article – How to: Export Your Data to CSV from a Database

SpatialKey needs good CSV data to work against!  In many cases an organizations data is not in CSV files, but locked away in a database.

Most (if not all) databases have the capability of getting table based data into a CSV format, but they are all different in the process and complexity of doing so.  To help with this issue, I have written an article on SpatialKey.com explaining the process of exporting data from several well-known databases (Microsoft Access 2003, 2007, MySQL, Postgres, MS SQL Server, and Oracle) into a CSV file.

Hope it helps!

Read the article here: Article