[PIPELINE]

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[PIPELINE]

Kevin Bootz
Is anyone monitoring/developing with the commons pipeline project? The few entries I've seen in the archives and the sandbox are years old and aside from the NOAA website(Ken Tanaka?) there are scarce resources for learning the suite. Any homegrown examples out there that do not incorporate the digester?

Thanks
Reply | Threaded
Open this post in threaded view
|

RE: [PIPELINE]

southmark
Hi Kevin,

I'm using it. I have some pipelines that pull down data from the web and insert into a hibernate based database. I have some generic Stages that can group/ungroup objects and handle hibernate sessions and committing to the database.

Generic Stages: http://code.google.com/p/pubchemdb/source/browse/#svn%2Ftrunk%2FPubChemDB%2Fsrc%2Fmain%2Fjava%2Fedu%2Fscripps%2Ffl%2Fpipeline

App specific Stages: http://code.google.com/p/pubchemdb/source/browse/#svn%2Ftrunk%2FPubChemDB%2Fsrc%2Fmain%2Fjava%2Fedu%2Fscripps%2Ffl%2Fpubchem%2Fapp

Pipeline xml examples: http://code.google.com/p/pubchemdb/source/browse/#svn%2Ftrunk%2FPubChemDB%2Fsrc%2Fmain%2Fresources%2Fedu%2Fscripps%2Ffl%2Fpubchem%253Fstate%253Dclosed

Best,
~Mark.


-----Original Message-----
From: Kevin Bootz [mailto:[hidden email]]
Sent: Friday, January 21, 2011 9:27 AM
To: [hidden email]
Subject: [PIPELINE]

Is anyone monitoring/developing with the commons pipeline project? The few entries I've seen in the archives and the sandbox are years old and aside from the NOAA website(Ken Tanaka?) there are scarce resources for learning the suite. Any homegrown examples out there that do not incorporate the digester?

Thanks

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: [PIPELINE]

Gary Gregory
Hi Mark,

Just curious: have you considered using Spring Batch?

Gary Gregory
Senior Software Engineer
Rocket Software
3340 Peachtree Road, Suite 820 . Atlanta, GA 30326 . USA
Tel: +1.404.760.1560
Email: [hidden email]
Web: seagull.rocketsoftware.comĀ 



> -----Original Message-----
> From: Mark Southern [mailto:[hidden email]]
> Sent: Friday, January 21, 2011 09:37
> To: Commons Users List
> Subject: RE: [PIPELINE]
>
> Hi Kevin,
>
> I'm using it. I have some pipelines that pull down data from the web and
> insert into a hibernate based database. I have some generic Stages that can
> group/ungroup objects and handle hibernate sessions and committing to the
> database.
>
> Generic Stages:
> <a href="http://code.google.com/p/pubchemdb/source/browse/#svn%2Ftrunk%2FPubChemDB%2">http://code.google.com/p/pubchemdb/source/browse/#svn%2Ftrunk%2FPubChemDB%2
> Fsrc%2Fmain%2Fjava%2Fedu%2Fscripps%2Ffl%2Fpipeline
>
> App specific Stages:
> <a href="http://code.google.com/p/pubchemdb/source/browse/#svn%2Ftrunk%2FPubChemDB%2">http://code.google.com/p/pubchemdb/source/browse/#svn%2Ftrunk%2FPubChemDB%2
> Fsrc%2Fmain%2Fjava%2Fedu%2Fscripps%2Ffl%2Fpubchem%2Fapp
>
> Pipeline xml examples:
> <a href="http://code.google.com/p/pubchemdb/source/browse/#svn%2Ftrunk%2FPubChemDB%2">http://code.google.com/p/pubchemdb/source/browse/#svn%2Ftrunk%2FPubChemDB%2
> Fsrc%2Fmain%2Fresources%2Fedu%2Fscripps%2Ffl%2Fpubchem%253Fstate%253Dclosed
>
> Best,
> ~Mark.
>
>
> -----Original Message-----
> From: Kevin Bootz [mailto:[hidden email]]
> Sent: Friday, January 21, 2011 9:27 AM
> To: [hidden email]
> Subject: [PIPELINE]
>
> Is anyone monitoring/developing with the commons pipeline project? The few
> entries I've seen in the archives and the sandbox are years old and aside
> from the NOAA website(Ken Tanaka?) there are scarce resources for learning
> the suite. Any homegrown examples out there that do not incorporate the
> digester?
>
> Thanks
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: [PIPELINE]

southmark
Thanks for the reference. I haven't. I came across commons-pipeline, it was easy and I looked no further!

-----Original Message-----
From: Gary Gregory [mailto:[hidden email]]
Sent: Friday, January 21, 2011 9:44 AM
To: Commons Users List
Subject: RE: [PIPELINE]

Hi Mark,

Just curious: have you considered using Spring Batch?

Gary Gregory
Senior Software Engineer
Rocket Software
3340 Peachtree Road, Suite 820 . Atlanta, GA 30326 . USA
Tel: +1.404.760.1560
Email: [hidden email]
Web: seagull.rocketsoftware.comĀ 



> -----Original Message-----
> From: Mark Southern [mailto:[hidden email]]
> Sent: Friday, January 21, 2011 09:37
> To: Commons Users List
> Subject: RE: [PIPELINE]
>
> Hi Kevin,
>
> I'm using it. I have some pipelines that pull down data from the web and
> insert into a hibernate based database. I have some generic Stages that can
> group/ungroup objects and handle hibernate sessions and committing to the
> database.
>
> Generic Stages:
> <a href="http://code.google.com/p/pubchemdb/source/browse/#svn%2Ftrunk%2FPubChemDB%2">http://code.google.com/p/pubchemdb/source/browse/#svn%2Ftrunk%2FPubChemDB%2
> Fsrc%2Fmain%2Fjava%2Fedu%2Fscripps%2Ffl%2Fpipeline
>
> App specific Stages:
> <a href="http://code.google.com/p/pubchemdb/source/browse/#svn%2Ftrunk%2FPubChemDB%2">http://code.google.com/p/pubchemdb/source/browse/#svn%2Ftrunk%2FPubChemDB%2
> Fsrc%2Fmain%2Fjava%2Fedu%2Fscripps%2Ffl%2Fpubchem%2Fapp
>
> Pipeline xml examples:
> <a href="http://code.google.com/p/pubchemdb/source/browse/#svn%2Ftrunk%2FPubChemDB%2">http://code.google.com/p/pubchemdb/source/browse/#svn%2Ftrunk%2FPubChemDB%2
> Fsrc%2Fmain%2Fresources%2Fedu%2Fscripps%2Ffl%2Fpubchem%253Fstate%253Dclosed
>
> Best,
> ~Mark.
>
>
> -----Original Message-----
> From: Kevin Bootz [mailto:[hidden email]]
> Sent: Friday, January 21, 2011 9:27 AM
> To: [hidden email]
> Subject: [PIPELINE]
>
> Is anyone monitoring/developing with the commons pipeline project? The few
> entries I've seen in the archives and the sandbox are years old and aside
> from the NOAA website(Ken Tanaka?) there are scarce resources for learning
> the suite. Any homegrown examples out there that do not incorporate the
> digester?
>
> Thanks
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: [PIPELINE]

Kevin Bootz
In reply to this post by southmark
Thanks Mark,
    I've read several of your posts. Most recently those discussing contributions you were thinking of making.

-----Original Message-----
From: Mark Southern [mailto:[hidden email]]
Sent: Friday, January 21, 2011 9:37 AM
To: Commons Users List
Subject: RE: [PIPELINE]

Hi Kevin,

I'm using it. I have some pipelines that pull down data from the web and insert into a hibernate based database. I have some generic Stages that can group/ungroup objects and handle hibernate sessions and committing to the database.

Generic Stages: http://code.google.com/p/pubchemdb/source/browse/#svn%2Ftrunk%2FPubChemDB%2Fsrc%2Fmain%2Fjava%2Fedu%2Fscripps%2Ffl%2Fpipeline

App specific Stages: http://code.google.com/p/pubchemdb/source/browse/#svn%2Ftrunk%2FPubChemDB%2Fsrc%2Fmain%2Fjava%2Fedu%2Fscripps%2Ffl%2Fpubchem%2Fapp

Pipeline xml examples: http://code.google.com/p/pubchemdb/source/browse/#svn%2Ftrunk%2FPubChemDB%2Fsrc%2Fmain%2Fresources%2Fedu%2Fscripps%2Ffl%2Fpubchem%253Fstate%253Dclosed

Best,
~Mark.


-----Original Message-----
From: Kevin Bootz [mailto:[hidden email]]
Sent: Friday, January 21, 2011 9:27 AM
To: [hidden email]
Subject: [PIPELINE]

Is anyone monitoring/developing with the commons pipeline project? The few entries I've seen in the archives and the sandbox are years old and aside from the NOAA website(Ken Tanaka?) there are scarce resources for learning the suite. Any homegrown examples out there that do not incorporate the digester?

Thanks

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [PIPELINE]

Ken Tanaka
In reply to this post by Kevin Bootz
I sent this reply yesterday but haven't seen it show up yet, so I'm
resending this--hopefully this isn't a duplicate.

We still use the pipeline for several scientific data loading projects.
A number of which summarize the spatial geometry and insert records into
a geospatial database (Oracle) that backs our ArcIMS interactive maps. (
Multibeam Bathymetry is one example
http://map.ngdc.noaa.gov/website/mgg/multibeam/viewer.htm ) The pipeline
code has a few updates that haven't been integrated yet (including Mark
Southern's contributions), but there hasn't been much development here
other than keeping up to date with the current version of maven. I know
there are a number of alternatives out there, but the pipeline is fairly
easy to use since it's focus is narrow. Besides Spring Batch, maybe
Kepler <https://kepler-project.org/>, Lift (Scala) or Akka
<http://akka.io/> (Java and Scala) are worth looking into.

We still configure the pipeline with digester, but I'd be interested in
any examples of configuring the pipeline with Spring or Groovy.

-Ken

Kevin Bootz wrote:
> Is anyone monitoring/developing with the commons pipeline project? The few entries I've seen in the archives and the sandbox are years old and aside from the NOAA website(Ken Tanaka?) there are scarce resources for learning the suite. Any homegrown examples out there that do not incorporate the digester?
>
> Thanks
>
>  


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]