[text] Incorporating Bruno Kinoshita's work

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[text] Incorporating Bruno Kinoshita's work

Benedikt Ritter-4
Hi,

the git repo for [text] is ready and I've done the initial bootstraping
already. I've also created a new component in the SANDBOX jira project. The
first issue is to extract algorithms from [lang] [1]. I remember people
saying, that theere is code in codec too. Please feel free to create
tickets for this.

Bruno already has some code that may fit into [text] [2]. I've given it a
brief review an here are few things which caught my eye:

- Inclusion of Talend code into [text] is not possible (the is code
licensed by www.talend.com)
- spellchecker package: nice idea, which I haven't thought about before.
Further more I could imagine a hyphenation package. Both should be locale
dependend.
- Looking at EditDistance [3] I'm not sure we need T extends Number, if the
only possible values for T are Integer and Double. Maybe we only need an
IntegerEditDistance and a DoubleEditDistance.

Regarding the last point: I'm currently not fond that there is a common
interface fot EditingDistance algorithms. For example Levenshtein has the
optional threshold parameter, which Jaro-Winkler has not (at least judging
from the implementation in [lang]). Fuzzy Distance needs a locale for
uncapitalizing. I think finding an interface that fits them all will be
difficult to accomplish... But we'll see :-)

Regards,
Benedikt

[1] https://issues.apache.org/jira/browse/SANDBOX-483
[2]
https://github.com/kinow/text/tree/master/src/main/java/text/string_metric
[3]
https://github.com/kinow/text/blob/master/src/main/java/text/string_metric/EditDistance.java

--
http://people.apache.org/~britter/
http://www.systemoutprintln.de/
http://twitter.com/BenediktRitter
http://github.com/britter
Reply | Threaded
Open this post in threaded view
|

Re: [text] Incorporating Bruno Kinoshita's work

Bruno P. Kinoshita
Hi Benedikt!
Thanks for bootstrapping the project :)
> - spellchecker package: nice idea, which I haven't thought about before.
Further more I could imagine a hyphenation package. Both should be locale
dependend.
Great idea this hyphenation package, +1. I would already have a use case in a production tool that parses some crazy PDF's and uses OpenNLP. We have cases where we need to check hyphenation before tagging the words, and at the moment what we have is a not so elegant solution.

If we manage to create a common API for spellchecker we could, perhaps, create implementations that call hunspell and jazzy, that have artefacts in maven central and are fairly easy to use. Not sure if that fits in text, maybe only the common interface.

> - Looking at EditDistance [3] I'm not sure we need T extends Number, if the
only possible values for T are Integer and Double. Maybe we only need an
IntegerEditDistance and a DoubleEditDistance.
Could be. As the code was quickly written for a proof of concept for a customer, definitely there are parts that need further thinking. I'd fine with either T extends Number or IntegerEditDistance and DoubleEditDistance.

> Regarding the last point: I'm currently not fond that there is a common
interface fot EditingDistance algorithms. For example Levenshtein has the
optional threshold parameter, which Jaro-Winkler has not (at least judging
from the implementation in [lang]). Fuzzy Distance needs a locale for
uncapitalizing. I think finding an interface that fits them all will be
difficult to accomplish... But we'll see :-)
I had thought about just a marker interface. So I could write some code to scan the classpath looking for implementations of this interface and let the user decide which one to use for his data quality job (regardless of the different parameters used in each algorithm).

I shamelessly stole the name StringMetric from this Wikipedia article [1], but maybe we could find a better name for it?
Thanks again Benedikt!
Bruno
[1] http://en.wikipedia.org/wiki/String_metric

 
      From: Benedikt Ritter <[hidden email]>
 To: Commons Developers List <[hidden email]>
 Sent: Wednesday, November 12, 2014 10:34 AM
 Subject: [text] Incorporating Bruno Kinoshita's work
   
Hi,

the git repo for [text] is ready and I've done the initial bootstraping
already. I've also created a new component in the SANDBOX jira project. The
first issue is to extract algorithms from [lang] [1]. I remember people
saying, that theere is code in codec too. Please feel free to create
tickets for this.

Bruno already has some code that may fit into [text] [2]. I've given it a
brief review an here are few things which caught my eye:

- Inclusion of Talend code into [text] is not possible (the is code
licensed by www.talend.com)
- spellchecker package: nice idea, which I haven't thought about before.
Further more I could imagine a hyphenation package. Both should be locale
dependend.
- Looking at EditDistance [3] I'm not sure we need T extends Number, if the
only possible values for T are Integer and Double. Maybe we only need an
IntegerEditDistance and a DoubleEditDistance.

Regarding the last point: I'm currently not fond that there is a common
interface fot EditingDistance algorithms. For example Levenshtein has the
optional threshold parameter, which Jaro-Winkler has not (at least judging
from the implementation in [lang]). Fuzzy Distance needs a locale for
uncapitalizing. I think finding an interface that fits them all will be
difficult to accomplish... But we'll see :-)

Regards,
Benedikt

[1] https://issues.apache.org/jira/browse/SANDBOX-483
[2]
https://github.com/kinow/text/tree/master/src/main/java/text/string_metric
[3]
https://github.com/kinow/text/blob/master/src/main/java/text/string_metric/EditDistance.java

--
http://people.apache.org/~britter/
http://www.systemoutprintln.de/
http://twitter.com/BenediktRitter
http://github.com/britter


   
 
Reply | Threaded
Open this post in threaded view
|

Re: [text] Incorporating Bruno Kinoshita's work

Emmanuel Bourg-3
In reply to this post by Benedikt Ritter-4
Le 12/11/2014 13:34, Benedikt Ritter a écrit :

> the git repo for [text] is ready and I've done the initial bootstraping
> already. I've also created a new component in the SANDBOX jira project. The
> first issue is to extract algorithms from [lang] [1]. I remember people
> saying, that theere is code in codec too. Please feel free to create
> tickets for this.

I think a PROPOSAL.html file defining the scope and goals of the
component would a good idea.

> - Inclusion of Talend code into [text] is not possible (the is code
> licensed by www.talend.com)

What is this code about?

> - spellchecker package: nice idea, which I haven't thought about before.
> Further more I could imagine a hyphenation package. Both should be locale
> dependend.

I may be wrong, but I'm under the impression a full spellchecking API is
probably too big for a small utility component.

Emmanuel Bourg


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [text] Incorporating Bruno Kinoshita's work

Bruno P. Kinoshita
Hi ebourg!

>I think a PROPOSAL.html file defining the scope and goals of thecomponent would a good idea.
+1
> > - Inclusion of Talend code into [text] is not possible (the is code
> > licensed by www.talend.com)
>
> What is this code about?
Talend Open Studio [1]. More specifically, part of the solution that provides a mechanism for combining multiple algorithms and uses probability, weights and thresholds for comparing attributes.

I started using Talend Open Studio, but have already switched to Duke [2], which is Apache License, has some algorithms already implemented [3] and includes the probabilistic methods as well.
Probably it's a better idea to use Duke's code, in case we decide to include the probabilistic method.

[1] https://www.talend.com/products/talend-open-studio
[2] https://github.com/larsga/Duke/
[3] https://github.com/larsga/Duke/tree/master/src/main/java/no/priv/garshol/duke/comparators


      From: Emmanuel Bourg <[hidden email]>
 To: Commons Developers List <[hidden email]>
 Sent: Wednesday, November 12, 2014 11:34 AM
 Subject: Re: [text] Incorporating Bruno Kinoshita's work
   
Le 12/11/2014 13:34, Benedikt Ritter a écrit :

> the git repo for [text] is ready and I've done the initial bootstraping
> already. I've also created a new component in the SANDBOX jira project. The
> first issue is to extract algorithms from [lang] [1]. I remember people
> saying, that theere is code in codec too. Please feel free to create
> tickets for this.

I think a PROPOSAL.html file defining the scope and goals of the
component would a good idea.

> - Inclusion of Talend code into [text] is not possible (the is code
> licensed by www.talend.com)

What is this code about?



> - spellchecker package: nice idea, which I haven't thought about before.
> Further more I could imagine a hyphenation package. Both should be locale
> dependend.

I may be wrong, but I'm under the impression a full spellchecking API is
probably too big for a small utility component.

Emmanuel Bourg


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



   
Reply | Threaded
Open this post in threaded view
|

Re: [text] Incorporating Bruno Kinoshita's work

Benedikt Ritter-4
In reply to this post by Emmanuel Bourg-3
Hello Emmanuel

2014-11-12 14:34 GMT+01:00 Emmanuel Bourg <[hidden email]>:

> Le 12/11/2014 13:34, Benedikt Ritter a écrit :
>
> > the git repo for [text] is ready and I've done the initial bootstraping
> > already. I've also created a new component in the SANDBOX jira project.
> The
> > first issue is to extract algorithms from [lang] [1]. I remember people
> > saying, that theere is code in codec too. Please feel free to create
> > tickets for this.
>
> I think a PROPOSAL.html file defining the scope and goals of the
> component would a good idea.
>

Yes, I'll work on that as soon as I have the time.


>
> > - Inclusion of Talend code into [text] is not possible (the is code
> > licensed by www.talend.com)
>
> What is this code about?
>

I don't know. It's some code that Bruno used in hier project. I don't think
we can use it.


>
> > - spellchecker package: nice idea, which I haven't thought about before.
> > Further more I could imagine a hyphenation package. Both should be locale
> > dependend.
>
> I may be wrong, but I'm under the impression a full spellchecking API is
> probably too big for a small utility component.
>

Yes you're right. I think it would be best to start up with the stuff we
already have. I already have a local branch (yeah, git!) for incorporating
the three algorithms from [lang].  We'll see were we go from there.

Benedikt


>
> Emmanuel Bourg
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
http://people.apache.org/~britter/
http://www.systemoutprintln.de/
http://twitter.com/BenediktRitter
http://github.com/britter
Reply | Threaded
Open this post in threaded view
|

Re: [text] Incorporating Bruno Kinoshita's work

J.Pietschmann
In reply to this post by Bruno P. Kinoshita
Hi all,

> Great idea this hyphenation package, +1.

There is a hyphenation package in Apache FOP. Whoever goes to work
on a similar package in [text] should have a look there.

Regards
J.Pietschmann

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]