Point of clarification

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Point of clarification

TheSteve0
Hey all:
Used Lucene was back in 2003 and saw all the rumbings about spatial being added but never had time to come back to lucene and explore it.
It seems like there are a couple of projects right now adding spatial to Lucene.

1) The spatial4j project. This seems to be the "official" way forward for spatial integration into lucene. It seems to support basic spatial operations on all geometries except polygons (unless I use JTS). I am having a hard time finding doc on how to use the libraries, especially with examples.

2) There is an alternate spatial implementation from Hibernate Search:
http://docs.jboss.org/hibernate/search/4.2/reference/en-US/html_single/#spatial

This appears not to be using the spatial4j functionality but instead uses its own spatial implementation. It also seems to be geared specifically to supporting point radius searching.


Is my read of the situation correct?
If not, then please update my understanding. I am about to write a blog post about lucene's spatial capabilities and I want to make sure I get this right.

Thanks
Steve
Developer Evangelist (OpenShift)
Reply | Threaded
Open this post in threaded view
|

Re: [Dev] Point of clarification

dsmiley
Hi Steve!

On Feb 6, 2013, at 10:41 PM, TheSteve0 <[hidden email]> wrote:

> Hey all:
> Used Lucene was back in 2003 and saw all the rumbings about spatial being
> added but never had time to come back to lucene and explore it.
> It seems like there are a couple of projects right now adding spatial to
> Lucene.

Things have changed!  That code has been retired.

>
> 1) The spatial4j project. This seems to be the "official" way forward for
> spatial integration into lucene. It seems to support basic spatial
> operations on all geometries except polygons (unless I use JTS). I am having
> a hard time finding doc on how to use the libraries, especially with
> examples.

Spatial4j is only about shapes.  In retrospect, "Shape4j" might have been an even better name.  So if you are going to work on new or improved shapes, then you're at the right place.  If you want to index and search these shapes in Lucene, then the Lucene spatial module is where that code is.  Lucene spatial uses Spatial4j.  Your comment on JTS is right.

There does need to be a Spatial4j quick start example that ties everything together.  Absent that, the javadocs are pervasive and the tests are comprehensive.

> 2) There is an alternate spatial implementation from Hibernate Search:
> http://docs.jboss.org/hibernate/search/4.2/reference/en-US/html_single/#spatial
>
> This appears not to be using the spatial4j functionality but instead uses
> its own spatial implementation. It also seems to be geared specifically to
> supporting point radius searching.

Thank you for bringing this to my attention; I just read it.  I wonder when this was introduced.  It has two strategies: A simple two-numeric field approach (which is akin to Lucene 4 spatial's PointVectorStrategy), and a more scalable quad-tree.  There is more than one way to go about implementing a quad-tree in Lucene, and Hibernate-spatial is different than what Lucene 4 spatial has. I like its use of an equal area projection; I've planned to get around to that some day but I don't think it will account for much of a performance gain.  It does appear that the search in hibernate-spatial is limited to a circle -- that's unfortunate.  And it also appears to be limited to indexing a single point per field in the document, not variable and not other shapes -- Lucene 4 spatial supports all that.  It does have a decent approach to distance sorting -- that is still a bit of an achilles heal to PrefixTreeStrategy.  It works but the implementation is a memory pig and not NRT search
  friendly.

I can't say there's anything I intend on borrowing from Hibernate spatial.  I should check out the code sometime to be sure.  Benchmarks would be the true test but absent that I'm abundantly confident as usual :-P that RecursivePrefixTreeStrategy (RPTS) will outperform it, based on a few things I see Hibernate Spatial is doing.  One thing is that it puts the levels into different fields, but that reduces data locality. RPTS uses one field with a single pass algorithm.  Secondly it generates a potentially large term query, whereas RPTS is a recursive divide-and-conquor algorithm that halts if there is no data in a grid cell.  It also switches to a faster scanning mode at lower depths.  Thirdly, Hibernate-spatial's last search stage will validate every point to ensure it's in the query shape, whereas RPTS only does this at the edge grid cells, not the ones strictly within the query shape.  Finally, Chris did some basic tests and it showed that a 32-ary grid (Geohash) performed
 better than a quad (4) -- both of which are available in Lucene 4 spatial.

> Is my read of the situation correct?
> If not, then please update my understanding. I am about to write a blog post
> about lucene's spatial capabilities and I want to make sure I get this
> right.

Cool, please let me know when it's published.  And if you'd like me to review your post prior to publication then I'd be happy to.

It wouldn't hurt to look at the Spatial4j roadmap proposal to see where I'm headed:
http://spatial4j.16575.n6.nabble.com/Dev-Spatial4j-Roadmap-td5001111.html
The main theme is less dependency on JTS by having our own WKT parser and more shape implementations (but alas, not yet polygon).

You may find the documentation of the Solr side of the Lucene spatial module worth looking at, as its basically a thin adapter on top of Lucene spatial: http://wiki.apache.org/solr/SolrAdaptersForLuceneSpatial4

The unofficial roadmap for Lucene spatial is that I'm working on Contains and Within spatial predicates, mostly applicable to indexed non-point shapes.  And then I'm doing exact (not grid-approximated) search for all the predicates.  And for Solr, it could use a proper query parser rather than the field query approach done now.

Cheers,
        ~ David
_______________________________________________
dev mailing list
[hidden email]
http://lists.spatial4j.com/listinfo.cgi/dev-spatial4j.com
Reply | Threaded
Open this post in threaded view
|

Re: [Dev] Point of clarification

dsmiley
In reply to this post by TheSteve0

On Wed, Feb 6, 2013 at 10:41 PM, TheSteve0 <[hidden email]> wrote:
I am having
a hard time finding doc on how to use the libraries, especially with
examples.

By the way, you should take a look at SpatialExample.java in Lucene spatial:
That exists for the express purpose of showing how to use Lucene spatial, which of course involves Spatial4j.

_______________________________________________
dev mailing list
[hidden email]
http://lists.spatial4j.com/listinfo.cgi/dev-spatial4j.com
Reply | Threaded
Open this post in threaded view
|

Re: Point of clarification

panpan2523
This post has NOT been accepted by the mailing list yet.
In reply to this post by TheSteve0
High quality is always together with high price. So, someone may be afraid of that they cannot afford it. Dear friends, never mind it. Now the entire Toms shoes sale is on a huge discount. These cheap http://buytomshere.com/goods-16-Toms-Girl-Canvas-White-Shoes-With-Unque-Design.html  shoes are also made of high quality and nice appearance. Go to the Toms shoes online store and choose your favorite one. And you will be surprised at the high quality and perfect service!
Reply | Threaded
Open this post in threaded view
|

Re: [Dev] Point of clarification

TheSteve0
In reply to this post by dsmiley
Hey Dave:
So I have been speaking to Vivid solutions. Would you be interested in talking to them about the value of them BSD licensing JTS?
Has there been any progress. I would really like to show this at FOSS4G NA.
Thanks
Steve

On 02/10/2013 08:47 PM, dsmiley [via Spatial4j] wrote:

On Wed, Feb 6, 2013 at 10:41 PM, TheSteve0 <[hidden email]> wrote:
I am having
a hard time finding doc on how to use the libraries, especially with
examples.

By the way, you should take a look at SpatialExample.java in Lucene spatial:
That exists for the express purpose of showing how to use Lucene spatial, which of course involves Spatial4j.

_______________________________________________
dev mailing list
[hidden email]
http://lists.spatial4j.com/listinfo.cgi/dev-spatial4j.com



If you reply to this email, your message will be added to the discussion below:
http://spatial4j.16575.n6.nabble.com/Point-of-clarification-tp5001136p5001138.html
To unsubscribe from Point of clarification, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: [Dev] Point of clarification

dsmiley
Hi Steve,

In the past, Martin has already expressed interest to me in licensing JTS in something more permissive.  It's up to him of course but I'll bet there are some obstacles that make it not so easy as it is for smaller projects which less history as JTS.  You could ask him yourself.  FYI I'm on the JTS mailing list.  If you're just interested in polygons, then FYI I've been encouraging/prodding someone at Raytheon to open-source his geodetic polygon code.  By geodetic I mean surface-of-a-sphere polygon, not JTS's 2D.  I'm more optimistic now than ever but I'm still not holding my breath.

RE FOSS4G NA, that's a conference I should make an effort to go to and present at.  Maybe next year when Spatial4j/Lucene-spatial is even further along.  But already, I believe there is enough evidence to make the bold claim that Lucene/Solr spatial has more advanced spatial support than any other search/NoSQL platform (as of v4.3).

~ David


On Thu, Apr 25, 2013 at 11:21 AM, TheSteve0 <[hidden email]> wrote:
Hey Dave:
So I have been speaking to Vivid solutions. Would you be interested in talking to them about the value of them BSD licensing JTS?
Has there been any progress. I would really like to show this at FOSS4G NA.
Thanks
Steve

On 02/10/2013 08:47 PM, dsmiley [via Spatial4j] wrote:

On Wed, Feb 6, 2013 at 10:41 PM, TheSteve0 <[hidden email]> wrote:
I am having
a hard time finding doc on how to use the libraries, especially with
examples.

By the way, you should take a look at SpatialExample.java in Lucene spatial:
That exists for the express purpose of showing how to use Lucene spatial, which of course involves Spatial4j.

_______________________________________________
dev mailing list
[hidden email]
http://lists.spatial4j.com/listinfo.cgi/dev-spatial4j.com



If you reply to this email, your message will be added to the discussion below:
http://spatial4j.16575.n6.nabble.com/Point-of-clarification-tp5001136p5001138.html
To unsubscribe from Point of clarification, click here.
NAML



View this message in context: Re: [Dev] Point of clarification
Sent from the Spatial4j mailing list archive at Nabble.com.

_______________________________________________
dev mailing list
[hidden email]
http://lists.spatial4j.com/listinfo.cgi/dev-spatial4j.com



_______________________________________________
dev mailing list
[hidden email]
http://lists.spatial4j.com/listinfo.cgi/dev-spatial4j.com
Reply | Threaded
Open this post in threaded view
|

Re: [Dev] Point of clarification

TheSteve0
Turns out Martin doesn't own the code - Vivid does. But we are trying to get some people together to talk to Vivid as a group to say "here is why BSD would really help our projects". Right now I don't think they see much to gain from going to BSD for the hassle it would take them.

So there are my thoughts...


On Fri, Apr 26, 2013 at 12:56 AM, dsmiley [via Spatial4j] <[hidden email]> wrote:
Hi Steve,

In the past, Martin has already expressed interest to me in licensing JTS in something more permissive.  It's up to him of course but I'll bet there are some obstacles that make it not so easy as it is for smaller projects which less history as JTS.  You could ask him yourself.  FYI I'm on the JTS mailing list.  If you're just interested in polygons, then FYI I've been encouraging/prodding someone at Raytheon to open-source his geodetic polygon code.  By geodetic I mean surface-of-a-sphere polygon, not JTS's 2D.  I'm more optimistic now than ever but I'm still not holding my breath.

RE FOSS4G NA, that's a conference I should make an effort to go to and present at.  Maybe next year when Spatial4j/Lucene-spatial is even further along.  But already, I believe there is enough evidence to make the bold claim that Lucene/Solr spatial has more advanced spatial support than any other search/NoSQL platform (as of v4.3).

~ David


On Thu, Apr 25, 2013 at 11:21 AM, TheSteve0 <[hidden email]> wrote:
Hey Dave:
So I have been speaking to Vivid solutions. Would you be interested in talking to them about the value of them BSD licensing JTS?
Has there been any progress. I would really like to show this at FOSS4G NA.
Thanks
Steve

On 02/10/2013 08:47 PM, dsmiley [via Spatial4j] wrote:

On Wed, Feb 6, 2013 at 10:41 PM, TheSteve0 <[hidden email]> wrote:
I am having
a hard time finding doc on how to use the libraries, especially with
examples.

By the way, you should take a look at SpatialExample.java in Lucene spatial:
That exists for the express purpose of showing how to use Lucene spatial, which of course involves Spatial4j.

_______________________________________________
dev mailing list
[hidden email]
http://lists.spatial4j.com/listinfo.cgi/dev-spatial4j.com



If you reply to this email, your message will be added to the discussion below:
http://spatial4j.16575.n6.nabble.com/Point-of-clarification-tp5001136p5001138.html
To unsubscribe from Point of clarification, click here.
NAML



View this message in context: Re: [Dev] Point of clarification
Sent from the Spatial4j mailing list archive at Nabble.com.

_______________________________________________
dev mailing list
[hidden email]
http://lists.spatial4j.com/listinfo.cgi/dev-spatial4j.com



_______________________________________________
dev mailing list
[hidden email]
http://lists.spatial4j.com/listinfo.cgi/dev-spatial4j.com



If you reply to this email, your message will be added to the discussion below:
http://spatial4j.16575.x6.nabble.com/Point-of-clarification-tp5001136p5001150.html
To unsubscribe from Point of clarification, click here.
NAML