[DISCUSS] Jenkins Pipeline DSL

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[DISCUSS] Jenkins Pipeline DSL

Benedikt Ritter-4
Hello,

currently we define our Jenkins job through the Jenkins UI. The problem
with this is, that there is no connection between the source code and the
way it is build. The Build job configuration is versioned separately from
the source code in Jenkins it self.

With the Jenkins Pipeline Plugin [1] it is possible to define Jenkins Jobs
in the source repository using a (groovy based) Job DSL [2]. So rather than
clicking through the Jenkins UI, one just writes down what jenkins should
to and checks it into version control.

The cool part is, that jenkins is than able to recreate the build pipeline
for every branch. This comes in handy when working with a gitflow like
development process, where there are several branches being changed.

I'd like to setup a PoC for the Commons Lang project showing, how this
looks like in real life. The PoC can than be adopted by other components,
if they wish.

Please let me know if you have objections.

Regards,
Benedikt

[1] https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin
[2] https://jenkins.io/solutions/pipeline/
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Jenkins Pipeline DSL

Bruno P. Kinoshita-3
+1 for the PoC

We can achieve something similar to what we have in Travis-CI through the Pipeline Plug-in (née Workflow Plug-in).

Here are some examples of what the Jenkinsfile looks like. In the Jenkinsfile you define, as Benedikt said, your job configuration with the DSL. It is not very different than writing a Travis-CI file, and if you have Jenkins installed locally, you can play with the Jenkinsfile before pushing it.

https://github.com/GoogleCloudPlatform/continuous-deployment-on-kubernetes/blob/master/sample-app/Jenkinsfile

https://github.com/eddumelendez/spring-boot-cache-sample/blob/master/Jenkinsfile


You can ask for user input, and in certain steps of the pipeline, where checkpoints are created, it is possible to restart jobs from that checkpoint. It may be useful for jobs that take a long time to run.

These were the pros, there are a few cons too. Like having to learn the DSL; not all plug-ins are supported to be used within pipelines; and I think it is not as declarative as Travis-CI, and can get messy. But I still like the idea of using the Pipeline Plug-in.

Digressing a little bit... the plug-in has a continuation passing style implementation for the pipelines/checkpoints/context/etc (you can read more about it here [1]). When I read the cps implementation in Jenkins, I recalled reading something similar in a commons component [2] Check the Team members of that module too :) I remember adding to my todo-list to check if Taverna workflows could benefit of continuations, and now Taverna is incubating in ASF. Might check that out some rainy day.

[1] https://github.com/jenkinsci/workflow-cps-plugin#technical-design


[2] https://commons.apache.org/sandbox/commons-javaflow/

>________________________________
> From: Benedikt Ritter <[hidden email]>
>To: Commons Developers List <[hidden email]>
>Sent: Saturday, 26 November 2016 11:18 PM
>Subject: [DISCUSS] Jenkins Pipeline DSL
>
>
>Hello,
>
>currently we define our Jenkins job through the Jenkins UI. The problem
>with this is, that there is no connection between the source code and the
>way it is build. The Build job configuration is versioned separately from
>the source code in Jenkins it self.
>
>With the Jenkins Pipeline Plugin [1] it is possible to define Jenkins Jobs
>in the source repository using a (groovy based) Job DSL [2]. So rather than
>clicking through the Jenkins UI, one just writes down what jenkins should
>to and checks it into version control.
>
>The cool part is, that jenkins is than able to recreate the build pipeline
>for every branch. This comes in handy when working with a gitflow like
>development process, where there are several branches being changed.
>
>I'd like to setup a PoC for the Commons Lang project showing, how this
>looks like in real life. The PoC can than be adopted by other components,
>if they wish.
>
>Please let me know if you have objections.
>
>Regards,
>Benedikt
>
>[1] https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin
>[2] https://jenkins.io/solutions/pipeline/
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Jenkins Pipeline DSL

sebb-2-2
In reply to this post by Benedikt Ritter-4
Seems OK to me.

However I think the script should be added to its own separate SVN/Git
branch (not trunk/master).
It's not a part of the source release per se, and as you write, it can
be applied to multiple branches.
Having it in the main source trees would be confusing, and more copies
to merge when updates are needed.

On 26 November 2016 at 10:18, Benedikt Ritter <[hidden email]> wrote:

> Hello,
>
> currently we define our Jenkins job through the Jenkins UI. The problem
> with this is, that there is no connection between the source code and the
> way it is build. The Build job configuration is versioned separately from
> the source code in Jenkins it self.
>
> With the Jenkins Pipeline Plugin [1] it is possible to define Jenkins Jobs
> in the source repository using a (groovy based) Job DSL [2]. So rather than
> clicking through the Jenkins UI, one just writes down what jenkins should
> to and checks it into version control.
>
> The cool part is, that jenkins is than able to recreate the build pipeline
> for every branch. This comes in handy when working with a gitflow like
> development process, where there are several branches being changed.
>
> I'd like to setup a PoC for the Commons Lang project showing, how this
> looks like in real life. The PoC can than be adopted by other components,
> if they wish.
>
> Please let me know if you have objections.
>
> Regards,
> Benedikt
>
> [1] https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin
> [2] https://jenkins.io/solutions/pipeline/

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Jenkins Pipeline DSL

garydgregory
That sounds even more complicated to deal with :-( I would have to fiddle
with branches just to make a Jenkins build change? Is there a precedent for
such a set up?

I do like the idea of having the Jenkins build file right there like Travis
CI but it is slightly worrisome that we have more files to keep in sync.
Nothing to do except proposing a POM based standard.

The nice thing about .travis.yml is that it is simple for plain cases like
for most Commons components.

Gary

On Nov 26, 2016 3:19 AM, "sebb" <[hidden email]> wrote:

> Seems OK to me.
>
> However I think the script should be added to its own separate SVN/Git
> branch (not trunk/master).
> It's not a part of the source release per se, and as you write, it can
> be applied to multiple branches.
> Having it in the main source trees would be confusing, and more copies
> to merge when updates are needed.
>
> On 26 November 2016 at 10:18, Benedikt Ritter <[hidden email]> wrote:
> > Hello,
> >
> > currently we define our Jenkins job through the Jenkins UI. The problem
> > with this is, that there is no connection between the source code and the
> > way it is build. The Build job configuration is versioned separately from
> > the source code in Jenkins it self.
> >
> > With the Jenkins Pipeline Plugin [1] it is possible to define Jenkins
> Jobs
> > in the source repository using a (groovy based) Job DSL [2]. So rather
> than
> > clicking through the Jenkins UI, one just writes down what jenkins should
> > to and checks it into version control.
> >
> > The cool part is, that jenkins is than able to recreate the build
> pipeline
> > for every branch. This comes in handy when working with a gitflow like
> > development process, where there are several branches being changed.
> >
> > I'd like to setup a PoC for the Commons Lang project showing, how this
> > looks like in real life. The PoC can than be adopted by other components,
> > if they wish.
> >
> > Please let me know if you have objections.
> >
> > Regards,
> > Benedikt
> >
> > [1] https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin
> > [2] https://jenkins.io/solutions/pipeline/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Jenkins Pipeline DSL

sebb-2-2
On 26 November 2016 at 15:45, Gary Gregory <[hidden email]> wrote:
> That sounds even more complicated to deal with :-( I would have to fiddle
> with branches just to make a Jenkins build change? Is there a precedent for
> such a set up?

Huh?

Surely the Jenkins setup script is independent of the build.

The precedent is that Jenkins is independently set up manually at
present; AIUI the difference is that one would use a script in SVN/Git
instead.
I assume this would only have to be done when the script changed or
when you wanted to set up a new build.

This is akin to running the Commons Build plugin.
That is in a separate part of the repo, but that makes it *less* complicated.
Imagine what it would be like if every branch had its own (slightly
different) copy...


> I do like the idea of having the Jenkins build file right there like Travis
> CI but it is slightly worrisome that we have more files to keep in sync.
> Nothing to do except proposing a POM based standard.
>
> The nice thing about .travis.yml is that it is simple for plain cases like
> for most Commons components.
>
> Gary
>
> On Nov 26, 2016 3:19 AM, "sebb" <[hidden email]> wrote:
>
>> Seems OK to me.
>>
>> However I think the script should be added to its own separate SVN/Git
>> branch (not trunk/master).
>> It's not a part of the source release per se, and as you write, it can
>> be applied to multiple branches.
>> Having it in the main source trees would be confusing, and more copies
>> to merge when updates are needed.
>>
>> On 26 November 2016 at 10:18, Benedikt Ritter <[hidden email]> wrote:
>> > Hello,
>> >
>> > currently we define our Jenkins job through the Jenkins UI. The problem
>> > with this is, that there is no connection between the source code and the
>> > way it is build. The Build job configuration is versioned separately from
>> > the source code in Jenkins it self.
>> >
>> > With the Jenkins Pipeline Plugin [1] it is possible to define Jenkins
>> Jobs
>> > in the source repository using a (groovy based) Job DSL [2]. So rather
>> than
>> > clicking through the Jenkins UI, one just writes down what jenkins should
>> > to and checks it into version control.
>> >
>> > The cool part is, that jenkins is than able to recreate the build
>> pipeline
>> > for every branch. This comes in handy when working with a gitflow like
>> > development process, where there are several branches being changed.
>> >
>> > I'd like to setup a PoC for the Commons Lang project showing, how this
>> > looks like in real life. The PoC can than be adopted by other components,
>> > if they wish.
>> >
>> > Please let me know if you have objections.
>> >
>> > Regards,
>> > Benedikt
>> >
>> > [1] https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin
>> > [2] https://jenkins.io/solutions/pipeline/
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]