[Dev] Status, and next steps

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[Dev] Status, and next steps

dsmiley
Presently, the project isn't build-able because spatial4j-solr & spatial4j-demo are out of sync with changes needed in Lucene spatial trunk to be compatible with the latest spatial4j-core. On top of this, there are some small changes in Solr trunk that are not yet adjusted for here.

One step I would like to do now, is to get SOLR-3304 "Add Solr support for the new Lucene spatial module" done as soon as possible.  To do so, I think we should take a spatial4j-solr snapshot dated at a time that is compatible with the current Lucene-spatial module.  Some small changes may be needed to address changes in Solr since then. This can probably just be a patch; it shouldn't be that big. We can then iterate from there.

Secondly, how can we address the problem of making improvements to spatial4j while simultaneously observing its effects in Lucene/Solr?  It's not lost on me that this is a problem of our creation with spatial4j being here instead of in Lucene/Solr itself.  My proposal is to fork Lucene-Solr on github into the spatial4j account, and then work on Lucene/Solr updates there as needed to comply with the latest Spatial4j-core work.  The changes therein should primarily be limited to compatibility, with spatial4j-core.  Maybe new features could be developed there but that doesn't seem like a good idea.  JtsGeoStrategy, for example, can't be committed to Lucene-Solr itself due to licensing, it obviously doesn't belong in spatial4j-core.  I'm not sure I like the spatial4j-solr & spatial4j-demo being included together with spatial4j-core in the same repo anymore, but it's not a big deal and a change there wouldn't affect my proposal in this paragraph any way.

Comments please.

~ David
_______________________________________________
dev mailing list
[hidden email]
http://lists.spatial4j.com/listinfo.cgi/dev-spatial4j.com
Reply | Threaded
Open this post in threaded view
|

Re: [Dev] Status, and next steps

Chris Male
Hey David,

Thanks for bringing this issues up.

On Sun, May 13, 2012 at 3:41 PM, David Smiley <[hidden email]> wrote:
Presently, the project isn't build-able because spatial4j-solr & spatial4j-demo are out of sync with changes needed in Lucene spatial trunk to be compatible with the latest spatial4j-core. On top of this, there are some small changes in Solr trunk that are not yet adjusted for here.

I think as long as we are dependent on the 'latest' trunk build we're going to have these kinds of synchronisation issues.  Of course, once we've resolved what to do with spatial4j-solr and demo we won't have to worry.  So I agree it's best to get a march on with sorting them out.
 

One step I would like to do now, is to get SOLR-3304 "Add Solr support for the new Lucene spatial module" done as soon as possible.  To do so, I think we should take a spatial4j-solr snapshot dated at a time that is compatible with the current Lucene-spatial module.  Some small changes may be needed to address changes in Solr since then. This can probably just be a patch; it shouldn't be that big. We can then iterate from there.

I agree.  I think we should make a new lucene branch for SOLR-3304 and get the Solr support in there as soon as possible.  We can then tweak it and merge it back into trunk.
 

Secondly, how can we address the problem of making improvements to spatial4j while simultaneously observing its effects in Lucene/Solr?  It's not lost on me that this is a problem of our creation with spatial4j being here instead of in Lucene/Solr itself.  My proposal is to fork Lucene-Solr on github into the spatial4j account, and then work on Lucene/Solr updates there as needed to comply with the latest Spatial4j-core work.  The changes therein should primarily be limited to compatibility, with spatial4j-core.  

Unsurprisingly, I see it the other way around.  I think this is exactly why we need spatial4j inside Lucene.  I don't see the spatial module as just being glue code for spatial4j-core.  I see the relationship going both ways.  I am a Lucene user first and foremost and the fact that spatial4j makes my spatial searches work is an implementation detail, Lucene is where I enter the stack.

As such I don't like the idea of us forking Lucene on github.  I think we should bring the development together in Lucene until we feel that spatial4j code is more settled.  I don't like when other people do their Lucene dev on github either really, I think it ruins the transparency of our development that we do in Lucene.  Code from somewhere else that only a few people knew of suddenly appears in trunk.
 
Maybe new features could be developed there but that doesn't seem like a good idea.  JtsGeoStrategy, for example, can't be committed to Lucene-Solr itself due to licensing, it obviously doesn't belong in spatial4j-core.  I'm not sure I like the spatial4j-solr & spatial4j-demo being included together with spatial4j-core in the same repo anymore, but it's not a big deal and a change there wouldn't affect my proposal in this paragraph any way.

spatial4j-solr is just in the repo till we get its place in Solr sorted out, right? 

I don't know what spatial4j-demo does, is it the same as the old demo code? As in, it provided a UI to play with the spatial stuff via Solr? In which case I think it has a place in Solr as well as demo code.  If that isn't possible for licensing / political issues, then I think it should be spun off as a separate project.

Cheers
Chris
 

Comments please.

~ David
_______________________________________________
dev mailing list
[hidden email]
http://lists.spatial4j.com/listinfo.cgi/dev-spatial4j.com



--
Chris Male | Software Developer | DutchWorks | www.dutchworks.nl

_______________________________________________
dev mailing list
[hidden email]
http://lists.spatial4j.com/listinfo.cgi/dev-spatial4j.com
Reply | Threaded
Open this post in threaded view
|

Re: [Dev] Status, and next steps

dsmiley
Chris, thanks for your input.

On Sun, May 13, 2012 at 5:12 AM, Chris Male <[hidden email]> wrote:
Hey David,

Thanks for bringing this issues up.

On Sun, May 13, 2012 at 3:41 PM, David Smiley <[hidden email]> wrote:
Presently, the project isn't build-able because spatial4j-solr & spatial4j-demo are out of sync with changes needed in Lucene spatial trunk to be compatible with the latest spatial4j-core. On top of this, there are some small changes in Solr trunk that are not yet adjusted for here.

I think as long as we are dependent on the 'latest' trunk build we're going to have these kinds of synchronisation issues.  Of course, once we've resolved what to do with spatial4j-solr and demo we won't have to worry.  So I agree it's best to get a march on with sorting them out.
 

One step I would like to do now, is to get SOLR-3304 "Add Solr support for the new Lucene spatial module" done as soon as possible.  To do so, I think we should take a spatial4j-solr snapshot dated at a time that is compatible with the current Lucene-spatial module.  Some small changes may be needed to address changes in Solr since then. This can probably just be a patch; it shouldn't be that big. We can then iterate from there.

I agree.  I think we should make a new lucene branch for SOLR-3304 and get the Solr support in there as soon as possible.  We can then tweak it and merge it back into trunk. 

The Solr module as of the release of spatial4j-0.2 consists of 6 java class files, 2 test java class files, and 2 testing config files.  Do you think a branch or patch is best?  I doubt there will be much iteration prior to commit.

** In a day or two, I will initiate it. **
 
Secondly, how can we address the problem of making improvements to spatial4j while simultaneously observing its effects in Lucene/Solr?  It's not lost on me that this is a problem of our creation with spatial4j being here instead of in Lucene/Solr itself.  My proposal is to fork Lucene-Solr on github into the spatial4j account, and then work on Lucene/Solr updates there as needed to comply with the latest Spatial4j-core work.  The changes therein should primarily be limited to compatibility, with spatial4j-core.  

Unsurprisingly, I see it the other way around.  I think this is exactly why we need spatial4j inside Lucene.  I don't see the spatial module as just being glue code for spatial4j-core.  

I don't see it as glue code either.  The Lucene module is center stage to all this work.  The Solr module is just an adapter, and Spatial4j is essentially a useful utility library of spatial shapes that has nothing to do with Lucene.
 
I see the relationship going both ways.  I am a Lucene user first and foremost and the fact that spatial4j makes my spatial searches work is an implementation detail, Lucene is where I enter the stack.

As such I don't like the idea of us forking Lucene on github.  I think we should bring the development together in Lucene until we feel that spatial4j code is more settled.  I don't like when other people do their Lucene dev on github either really, I think it ruins the transparency of our development that we do in Lucene.  Code from somewhere else that only a few people knew of suddenly appears in trunk.

I agree that improvements to the Lucene spatial module and (once it gets into Solr trunk) the Solr parts should occur in a normal fashion to other features in Lucene/Solr -- in Apache JIRA and not here.  I disagree that work on github isn't transparent, but it is less visible because the community isn't there so I get your point.  As I said, the point of the fork is merely to capture trickle-down changes in Lucene/Solr in response to API changes in Spatial4j.  So it's not a typical project fork.  To illustrate the need for it right now, I'm willing to bet that Ryan has on his computer some changes to the Lucene spatial module to make it compatible with Spatial4j-trunk.  He must because I know he's made changes to Spatial4j-Solr and Spatial4j-Demo which are downstream from that.  I'd like to see Ryan's changes so that I can test out the code overall, try the demo, etc.  A Lucene-Solr fork accomplishes that goal.  Github makes it easy to do.

 
Maybe new features could be developed there but that doesn't seem like a good idea.

To further appease your concerns, I'll further state that it not only isn't a good idea but it shouldn't be done, IMO.  
 
 JtsGeoStrategy, for example, can't be committed to Lucene-Solr itself due to licensing, it obviously doesn't belong in spatial4j-core.  I'm not sure I like the spatial4j-solr & spatial4j-demo being included together with spatial4j-core in the same repo anymore, but it's not a big deal and a change there wouldn't affect my proposal in this paragraph any way.

spatial4j-solr is just in the repo till we get its place in Solr sorted out, right? 

Yes, with the exception of JtsGeoStrategy.  That will have to find a new home in a new module & downloadable artifact for those who don't mind JTS's license.  To better delineate Spatial4j-core as independent of Lucene/Solr, I suggest JtsGeoStrategy go into a new git repo under Spatial4j along with the demo.  
 
I don't know what spatial4j-demo does, is it the same as the old demo code? As in, it provided a UI to play with the spatial stuff via Solr?

Yes.
 
In which case I think it has a place in Solr as well as demo code.

That is an interesting idea.  Hmmm.  I don't want to bloat Solr with this though.  The web resources like OpenLayers could be linked to online sources.  I dunno about use of Wicket.
 
 If that isn't possible for licensing / political issues, then I think it should be spun off as a separate project.

That's my preference.

~ David 

_______________________________________________
dev mailing list
[hidden email]
http://lists.spatial4j.com/listinfo.cgi/dev-spatial4j.com
Reply | Threaded
Open this post in threaded view
|

Re: [Dev] Status, and next steps

Chris Male
Hi,

On Tue, May 15, 2012 at 4:02 PM, [hidden email] <[hidden email]> wrote:
Chris, thanks for your input.

On Sun, May 13, 2012 at 5:12 AM, Chris Male <[hidden email]> wrote:
Hey David,

Thanks for bringing this issues up.

On Sun, May 13, 2012 at 3:41 PM, David Smiley <[hidden email]> wrote:
Presently, the project isn't build-able because spatial4j-solr & spatial4j-demo are out of sync with changes needed in Lucene spatial trunk to be compatible with the latest spatial4j-core. On top of this, there are some small changes in Solr trunk that are not yet adjusted for here.

I think as long as we are dependent on the 'latest' trunk build we're going to have these kinds of synchronisation issues.  Of course, once we've resolved what to do with spatial4j-solr and demo we won't have to worry.  So I agree it's best to get a march on with sorting them out.
 

One step I would like to do now, is to get SOLR-3304 "Add Solr support for the new Lucene spatial module" done as soon as possible.  To do so, I think we should take a spatial4j-solr snapshot dated at a time that is compatible with the current Lucene-spatial module.  Some small changes may be needed to address changes in Solr since then. This can probably just be a patch; it shouldn't be that big. We can then iterate from there.

I agree.  I think we should make a new lucene branch for SOLR-3304 and get the Solr support in there as soon as possible.  We can then tweak it and merge it back into trunk. 

The Solr module as of the release of spatial4j-0.2 consists of 6 java class files, 2 test java class files, and 2 testing config files.  Do you think a branch or patch is best?  I doubt there will be much iteration prior to commit.

The reason I suggested the branch was what's our strategy for integrating with the existing spatial stuff in Solr? Are we simply ignoring it and adding more options for the user? If that's the case then I'm sure a patch is fine.  But if we're planning on integrating it all together you'd be surprised how messy the existing code is so a branch might make it easier to work through the problems.  I don't mind either way really, whatever you feel most comfortable with.
 

** In a day or two, I will initiate it. **
 
Secondly, how can we address the problem of making improvements to spatial4j while simultaneously observing its effects in Lucene/Solr?  It's not lost on me that this is a problem of our creation with spatial4j being here instead of in Lucene/Solr itself.  My proposal is to fork Lucene-Solr on github into the spatial4j account, and then work on Lucene/Solr updates there as needed to comply with the latest Spatial4j-core work.  The changes therein should primarily be limited to compatibility, with spatial4j-core.  

Unsurprisingly, I see it the other way around.  I think this is exactly why we need spatial4j inside Lucene.  I don't see the spatial module as just being glue code for spatial4j-core.  

I don't see it as glue code either.  The Lucene module is center stage to all this work.  The Solr module is just an adapter, and Spatial4j is essentially a useful utility library of spatial shapes that has nothing to do with Lucene.

Agreed.
 
 
I see the relationship going both ways.  I am a Lucene user first and foremost and the fact that spatial4j makes my spatial searches work is an implementation detail, Lucene is where I enter the stack.

As such I don't like the idea of us forking Lucene on github.  I think we should bring the development together in Lucene until we feel that spatial4j code is more settled.  I don't like when other people do their Lucene dev on github either really, I think it ruins the transparency of our development that we do in Lucene.  Code from somewhere else that only a few people knew of suddenly appears in trunk.

I agree that improvements to the Lucene spatial module and (once it gets into Solr trunk) the Solr parts should occur in a normal fashion to other features in Lucene/Solr -- in Apache JIRA and not here.  I disagree that work on github isn't transparent, but it is less visible because the community isn't there so I get your point.  As I said, the point of the fork is merely to capture trickle-down changes in Lucene/Solr in response to API changes in Spatial4j.  So it's not a typical project fork.  To illustrate the need for it right now, I'm willing to bet that Ryan has on his computer some changes to the Lucene spatial module to make it compatible with Spatial4j-trunk.  He must because I know he's made changes to Spatial4j-Solr and Spatial4j-Demo which are downstream from that.  I'd like to see Ryan's changes so that I can test out the code overall, try the demo, etc.  A Lucene-Solr fork accomplishes that goal.  Github makes it easy to do.

If Spatial4j wasn't hanging out there by itself we wouldn't have this problem since the Lucene, Solr and core spatial4j code would all be of the same version.  You could play with Ryan's changes because he'd make a patch and put it into JIRA.  Yes using SVN is a pain in terms of playing with changes, I'm well aware of that and battle it everyday but I don't agree with doing all the changes over in github just because it's easier.

Why do we even have the spatial module in Lucene at all? It seems the other easier alternative was that we kept it in github and targeted specific Lucene versions.  We added it to Lucene so we could reach the maximum number of users and get other devs involved but now we're going to do the development back in github away from the eyes of uses and other devs.  I don't like it.
 

 
Maybe new features could be developed there but that doesn't seem like a good idea.

To further appease your concerns, I'll further state that it not only isn't a good idea but it shouldn't be done, IMO.  

I see no difference between developing new features and tweaking existing code.
 
 
 JtsGeoStrategy, for example, can't be committed to Lucene-Solr itself due to licensing, it obviously doesn't belong in spatial4j-core.  I'm not sure I like the spatial4j-solr & spatial4j-demo being included together with spatial4j-core in the same repo anymore, but it's not a big deal and a change there wouldn't affect my proposal in this paragraph any way.

spatial4j-solr is just in the repo till we get its place in Solr sorted out, right? 

Yes, with the exception of JtsGeoStrategy.  That will have to find a new home in a new module & downloadable artifact for those who don't mind JTS's license.  To better delineate Spatial4j-core as independent of Lucene/Solr, I suggest JtsGeoStrategy go into a new git repo under Spatial4j along with the demo.  

We really should explore apache-extras for the JTS stuff. http://code.google.com/a/apache-extras.org/hosting/.
 
 
I don't know what spatial4j-demo does, is it the same as the old demo code? As in, it provided a UI to play with the spatial stuff via Solr?

Yes.
 
In which case I think it has a place in Solr as well as demo code.

That is an interesting idea.  Hmmm.  I don't want to bloat Solr with this though.  The web resources like OpenLayers could be linked to online sources.  I dunno about use of Wicket.

It doesn't need to be in Solr core, but I feel it belongs somewhere.  Demo code is important, I don't think it's bloat.  Wicket is Apache Wicket so there shouldn't be a problem?
 
 
 If that isn't possible for licensing / political issues, then I think it should be spun off as a separate project.

That's my preference.

More code spread over more repositories being developed by who knows who, battling synchronization issues.  I dunno man!
 

~ David 

_______________________________________________
dev mailing list
[hidden email]
http://lists.spatial4j.com/listinfo.cgi/dev-spatial4j.com




--
Chris Male | Software Developer | DutchWorks | www.dutchworks.nl

_______________________________________________
dev mailing list
[hidden email]
http://lists.spatial4j.com/listinfo.cgi/dev-spatial4j.com
Reply | Threaded
Open this post in threaded view
|

Re: [Dev] Status, and next steps

dsmiley
In reply to this post by dsmiley
I was actually mistaken:

On Sat, May 12, 2012 at 11:41 PM, David Smiley <[hidden email]> wrote:
Presently, the project isn't build-able because spatial4j-solr & spatial4j-demo are out of sync with changes needed in Lucene spatial trunk to be compatible with the latest spatial4j-core. On top of this, there are some small changes in Solr trunk that are not yet adjusted for here.

The project is build-able now as it was re-synced with trunk (thanks Ryan).  There are no spatial4j-core api changes, at least none that provoked a change to lucene-spatial-module.  I got confused in part because my IntelliJ oddly put some things in red even though it could build & run, and I had seen changes that I figured were incompatible but I was mistaken.

But this doesn't make the discussion here pointless at all because the API will change eventually (such as via the proj4j branch).

p.s. I'll get moving on adding SOLR-3304

~ David

_______________________________________________
dev mailing list
[hidden email]
http://lists.spatial4j.com/listinfo.cgi/dev-spatial4j.com