[VOTE] Release Apache Commons RNG 1.3 based on RC1

classic Classic list List threaded Threaded
28 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Apache Commons RNG 1.3 based on RC1

Alex Herbert

On 13/11/2019 11:38, Gilles Sadowski wrote:

> Hello.
>
> Le mar. 12 nov. 2019 à 17:41, Alex Herbert <[hidden email]> a écrit :
>>
>>
>>> On 12 Nov 2019, at 15:39, Gilles Sadowski <[hidden email]> wrote:
>>>
>>> Hello Alex.
>>>
>>> Le mar. 12 nov. 2019 à 16:15, Alex Herbert <[hidden email] <mailto:[hidden email]>> a écrit :
>>>>
>>>> On 12/11/2019 09:06, Gilles Sadowski wrote:
>>>>> 2019-11-12 2:28 UTC+01:00, Alex Herbert <[hidden email]>:
>>>>>>> On 11 Nov 2019, at 23:40, Gilles Sadowski <[hidden email]> wrote:
>>>>>>>
>>>>>>> Hi.
>>>>>>>
>>>>>>> Maybe I'm missing what the issues really are,
>>>>>> All empty japicmp reports on the site.
>>>>>> Some confusing empty Jacoco aggregate reports on the site.
>>>>>>
>>>>>>> so sorry if this top-posted
>>>>>>> reply is beside the points:
>>>>>>> 1. There always have been several issues with JapiCmp (I do not remember
>>>>>>> exactly which; it must be in the ML archive); it never worked with
>>>>>>> Commons
>>>>>>> RNG.  [As as been mentioned in some thread and on JIRA, it might be worth
>>>>>>> spending time (if any) setting up the tools provided by the "revapi"
>>>>>>> project.]
>>>>>>> For now, Clirr still works fine for [RNG] (IIUC because of its "old" JDK
>>>>>>> target)
>>>>>>> and there is no need to have JapiCmp.
>>>>>> I’ve got japicmp to work in master. Maybe old versions had problems that
>>>>>> have now been fixed.
>>>>> I seem to remember that it failed the build for release 1.0 because there
>>>>> was no version to compare with (and it couldn't be prevented to run using
>>>>> the CP setup - cf. below).
>>>> Looking at the documentation I think this problem has been fixed with
>>>> optional properties. The appropriate property is not used in CP but a
>>>> project could just set it to true and the build would not fail.
>>>>
>>>>>> I think commons-parent should not be setup to generated empty reports when
>>>>>> it is not included as a profile.
>>>>> +1
>>>>> That was one of the issue.  Such plugins are optionally activated by the
>>>>> presence of a file, and should not run if the file is not present.  The setup
>>>>> works for other tools but it didn't for JapiCmp.
>>>>>
>>>>>>> 2. IMHO, there is no need for Jacoco aggregate reports; each module has
>>>>>>> its
>>>>>>> own "plain" report, accessible under the module's "sub-site" along with
>>>>>>> all
>>>>>>> the other reports concerned with that particular body of code.  If
>>>>>>> designed
>>>>>>> correctly (and in order to work under JPMS), the modules must have
>>>>>>> acyclic dependencies, and it seems to me equally meaningless to have
>>>>>>> modules
>>>>>>> aggregate reports as to have aggregate reports of external dependencies.
>>>>>> +1.
>>>>>>
>>>>>> I’ve disabled the aggregate report in RNG. I think it should be disabled in
>>>>>> commons-parent.
>>>>> + 1
>>>>> AFAICT, the latest CP enhanced automation does not take into account
>>>>> the "multi-module" maven feature.
>>>>> I had raised the question of why a "dist-archive" module is necessary:
>>>>> It seems to me that all the info being in the modules, the main POM
>>>>> should be able to generate, collect and "archive" all the artefacts under
>>>>> its own "target" directory.
>>>>> I've never dug very deep into maven, so I don't how whether it's possible
>>>>> or whether it's indeed to be done the (contorted, IMHO) way it is now.
>>>> I've tried to update RNG to work with japicmp conditionally.
>>>> Unfortunately once the maven plugin is included there is no way to
>>>> totally disable it. It will always put up an empty report in the site
>>>> generation.
>>>>
>>>> The fix was to locally change CP 49 (which RNG depends on) to move all
>>>> the configuration inside the profile that is activated using the file
>>>> 'src/site/resources/profile.japicmp'.
>>>>
>>>> A first fix with this present broke the RNG build on pre-Java8 JDKs. The
>>>> profile activation should also include activation if the JDK is 1.8+:
>>>>
>>>>        <activation>
>>>>          <jdk>[1.8,)</jdk>
>>>>          <file>
>>>> <exists>src/site/resources/profile.japicmp</exists>
>>>>          </file>
>>>>        </activation>
>>>>
>>>> I've rebuilt the report pages of the site using this set-up (fixed RNG,
>>>> fudged CP 49 locally). The JaCoCo aggregates are gone and the japicmp
>>>> reports are gone for all but:
>>>>
>>>> https://commons.apache.org/proper/commons-rng/commons-rng-client-api/japicmp.html
>>>>
>>>> https://commons.apache.org/proper/commons-rng/commons-rng-core/japicmp.html
>>>>
>>>> https://commons.apache.org/proper/commons-rng/commons-rng-simple/japicmp.html
>>>>
>>>> https://commons.apache.org/proper/commons-rng/commons-rng-sampling/japicmp.html
>>>>
>>>>
>>>> To fix the RNG build so that it can optionally include japicmp in the
>>>> report menus will require a change to the parent.
>>>>
>>>> However some projects may not be using
>>>> 'src/site/resources/profile.japicmp' to activate the profile. They may
>>>> be directly using <japicmp.skip>false</japicmp.skip>.
>>>>
>>>> So how to proceed with a fix for CP?
>>> If you found a fix, please apply it to CP and "release" v50.
>>> If there are issues, they will be fixed in 51+.
>> Q. Should this be run by the dev mailing list under the prefix [parent]?
> Or [CP] or maybe [All].
> Wouldn't hurt.

I have made changes to CP for jacoco and japicmp. They seem to be
non-destructive.

I am trying a fix to enable MathJax. It involves updating commons-skin
(last release in 2015) as that is where the header is set for all site
pages.

>
>> Looking at the site for the modules the top right icon is missing, e.g.
>>
>> https://commons.apache.org/proper/commons-rng/commons-rng-core/index.html <https://commons.apache.org/proper/commons-rng/commons-rng-core/index.html>
>>
>> It comes from ’src/site/site.xml’
>>
>> For all the modules this still requires images/commons_rng.small.png, which is missing.
>>
>> So:
>>
>> - Duplicate this image through the entire hierarchy
>> - Fix it to refer to the official image: https://commons.apache.org/proper/commons-rng/images/commons_rng.small.png <https://commons.apache.org/proper/commons-rng/images/commons_rng.small.png>
> +1 (the latter)
> [I've seen that it's done already.]

Yes. The maven site plugin tries to relativize absolute URLs that point
to the deploy site URL. So I've used this feature in the site.xml and
now the banner is fixed everywhere.

I still have to figure out how to post process the user guide (rng.apt)
to update the <pre> tags generated by doxia to <pre "prettyprint">. I
did this on the manually written pages and now code is rendered with
highlighting, e.g.

https://commons.apache.org/proper/commons-rng/commons-rng-simple/index.html

>
>> The link could also be made non-relative: https://commons.apache.org/proper/commons-rng/ <https://commons.apache.org/proper/commons-rng/>
>>
>> This would allow you to get back to the top level page from any of the modules pages. I don’t think this is possible at the moment.
> +1
>
> Thanks,
> Gilles
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[RNG] Site layout for modular components (Was: Release [...] RNG [...])

Gilles Sadowski-2
Hi.

Do you think that it would be feasible to have all those fixes applied
to other modular components (that were based on the [RNG] layout)
through a common layer (another POM) between those components
and "commons-parent"?

Regards,
Gilles

Le mer. 13 nov. 2019 à 12:53, Alex Herbert <[hidden email]> a écrit :
>
> [...]

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [RNG] Site layout for modular components (Was: Release [...] RNG [...])

Alex Herbert

On 13/11/2019 13:09, Gilles Sadowski wrote:
> Hi.
>
> Do you think that it would be feasible to have all those fixes applied
> to other modular components (that were based on the [RNG] layout)
> through a common layer (another POM) between those components
> and "commons-parent"?

Most of the fixes have been in the module site descriptors or items that
should be moved up to commons parent. It may be possible to put the site
descriptors into a parent. IIUC the site descriptors are merged together
before the maven site plugin creates the site. So the fix for the top
right banner could be moved into a common parent. Each child module
would also want to enumerate the past releases of the javadocs. Thus
they would require a site descriptor anyway and the banner fix would
only save 5 lines per site.xml for the banner.


I did a big change to the use of svn to checkout the current site. This
was required as the archived javadocs are not in a top level directory.
So each module has to be checked out, the archived javadocs set to be
excluded and then the rest of the site can be checked out.

Since this process is repeated for every module it can get very slow. I
changed the site checkout to copy the directory from the parent if it
was present. There was no solution I could find to fix this to run in a
single maven command as profile activation for all modules is done by
the reactor before any work is performed. Thus if the parent does the
checkout there is no way for all the child modules to know the parent
checkout exists in their profiles: when the profiles are initialised the
parent checkout may not exist.

There may be a better way to do this but it is all done in an antrun
plugin. Ant has limited if-else capability in the antrun plugin as it
only allows 1 <target> tag per execution. You require multiple <target>
tags in the same ant build to have conditional dependencies between
them, i.e. check for parent folder checkout and copy, otherwise do a svn
checkout.

I am going to try moving all this to a build.xml file and call that. It
should allow more powerful use of ant. It also separates the ant config
from the maven config.

If this works the build.xml for ant to checkout the site recursively
could be put into a parent.

I am still unclear on why this site checkout is required for all the
child modules. It is used in the maven-scm-publish-plugin but that
should be used on the top level module during a release process. So a
simpler fix is to not checkout the site in all the children.

A simple test would be to set the poms to not copy the directory for all
the child modules and do a dummy release.

It's a work in progress...


>
> Regards,
> Gilles
>
> Le mer. 13 nov. 2019 à 12:53, Alex Herbert <[hidden email]> a écrit :
>> [...]
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [RNG] Site layout for modular components (Was: Release [...] RNG [...])

Gilles Sadowski-2
Hello.

Le mer. 13 nov. 2019 à 18:29, Alex Herbert <[hidden email]> a écrit :

>
>
> On 13/11/2019 13:09, Gilles Sadowski wrote:
> > Hi.
> >
> > Do you think that it would be feasible to have all those fixes applied
> > to other modular components (that were based on the [RNG] layout)
> > through a common layer (another POM) between those components
> > and "commons-parent"?
>
> Most of the fixes have been in the module site descriptors or items that
> should be moved up to commons parent. It may be possible to put the site
> descriptors into a parent. IIUC the site descriptors are merged together
> before the maven site plugin creates the site. So the fix for the top
> right banner could be moved into a common parent. Each child module
> would also want to enumerate the past releases of the javadocs. Thus
> they would require a site descriptor anyway and the banner fix would
> only save 5 lines per site.xml for the banner.

Well, I was thinking of whether a multi-layered POM could be more
flexible and robust in handling the different type of components (e.g.
"multi-module" or not).

> I did a big change to the use of svn to checkout the current site. This
> was required as the archived javadocs are not in a top level directory.
> So each module has to be checked out, the archived javadocs set to be
> excluded and then the rest of the site can be checked out.

Sorry, I don't follow.
Didn't links to the Javadoc of previous versions exist prior to those
changes?

>
> Since this process is repeated for every module it can get very slow. I
> changed the site checkout to copy the directory from the parent if it
> was present. There was no solution I could find to fix this to run in a
> single maven command as profile activation for all modules is done by
> the reactor before any work is performed. Thus if the parent does the
> checkout there is no way for all the child modules to know the parent
> checkout exists in their profiles: when the profiles are initialised the
> parent checkout may not exist.
>
> There may be a better way to do this but it is all done in an antrun
> plugin. Ant has limited if-else capability in the antrun plugin as it
> only allows 1 <target> tag per execution. You require multiple <target>
> tags in the same ant build to have conditional dependencies between
> them, i.e. check for parent folder checkout and copy, otherwise do a svn
> checkout.
>
> I am going to try moving all this to a build.xml file and call that. It
> should allow more powerful use of ant. It also separates the ant config
> from the maven config.
>
> If this works the build.xml for ant to checkout the site recursively
> could be put into a parent.
>
> I am still unclear on why this site checkout is required for all the
> child modules. It is used in the maven-scm-publish-plugin but that
> should be used on the top level module during a release process. So a
> simpler fix is to not checkout the site in all the children.
>
> A simple test would be to set the poms to not copy the directory for all
> the child modules and do a dummy release.
>
> It's a work in progress...

I'm lost; what's the purpose?

Gilles

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [RNG] Site layout for modular components (Was: Release [...] RNG [...])

Alex Herbert


> On 13 Nov 2019, at 23:53, Gilles Sadowski <[hidden email]> wrote:
>
> Hello.
>
> Le mer. 13 nov. 2019 à 18:29, Alex Herbert <[hidden email] <mailto:[hidden email]>> a écrit :
>>
>>
>> On 13/11/2019 13:09, Gilles Sadowski wrote:
>>> Hi.
>>>
>>> Do you think that it would be feasible to have all those fixes applied
>>> to other modular components (that were based on the [RNG] layout)
>>> through a common layer (another POM) between those components
>>> and "commons-parent"?
>>
>> Most of the fixes have been in the module site descriptors or items that
>> should be moved up to commons parent. It may be possible to put the site
>> descriptors into a parent. IIUC the site descriptors are merged together
>> before the maven site plugin creates the site. So the fix for the top
>> right banner could be moved into a common parent. Each child module
>> would also want to enumerate the past releases of the javadocs. Thus
>> they would require a site descriptor anyway and the banner fix would
>> only save 5 lines per site.xml for the banner.
>
> Well, I was thinking of whether a multi-layered POM could be more
> flexible and robust in handling the different type of components (e.g.
> "multi-module" or not).

I think this would take a bit more reading on exactly how Maven thinks a multi-module project should work...

>
>> I did a big change to the use of svn to checkout the current site. This
>> was required as the archived javadocs are not in a top level directory.
>> So each module has to be checked out, the archived javadocs set to be
>> excluded and then the rest of the site can be checked out.
>
> Sorry, I don't follow.
> Didn't links to the Javadoc of previous versions exist prior to those
> changes?

Yes. Take a look at the pom.xml on line 464 for prepare-checkout.

This entire section was and is a mystery to me. I don’t know why it is required to create a copy of the site locally.

The previous code in this maven target was created on the assumption that it was a single module project and the archived javadocs for every previous version were in a javadocs directory under the top level directory. The same code is present in lang, io, text, etc.

The way it works for those projects is incorrect for a multi-module site as the archived javadocs are in a different place. It also is a target run in every module and so for a full site build with the examples modules you ended up with 14 full checkout copies of the RNG site, including the old archived javadocs.

Anyway I fixed it to work as it should. It checks out the site and ignores the old javadocs. I added a fix so child modules copy the parent site checkout. This only works if invoked in two stages on a clean git checkout:

mvn -N pre-site && mvn pre-site

This in itself is annoying as you cannot do:

git clone …
cd commons-rng && mvn site

Without the checkout of the site in each module. I think I can fix this by using an external ant build.xml file where you can do if-else logic. But I was wondering if I even had to. Perhaps the goal only needs to run in the parent, or the dist-archive module for a release.

What I would like to know is:

1. Do the child modules need this?
2. What is it used for? If it is only for a release then it should be in the release profile. Or maybe the commons-release plugin. Using ant to do this external execution is just poor (I spent a while fighting ant and it is not a programming language, so not suitable for the decisions required for the multi-module recursive processing).
3. The top level checkout is used in the release process for manually updating the site. But why all the others?

If I go to for example commons-rng-client-api and empty the site-content folder ‘mvn clean package site’ still creates a site that looks fine.

Another bug with multi-module builds is that the following reports are in each child module and are duplicates of that in the top level page:

Project information
- Team
- SCM
- Issue Management
- Mailing lists
- CI management
- Distribution management

Project reports
- jira

The ones in the project information menu are harmless and fast to create. The jira report takes a long time to generate. I think at least this one should be disabled in child modules.

My motivation is to reduce bloat and and speed up the process of a site build. It was annoying when doing the release as you use clean checkouts and the download of 14 copies of the full site was slow.

>
>>
>> Since this process is repeated for every module it can get very slow. I
>> changed the site checkout to copy the directory from the parent if it
>> was present. There was no solution I could find to fix this to run in a
>> single maven command as profile activation for all modules is done by
>> the reactor before any work is performed. Thus if the parent does the
>> checkout there is no way for all the child modules to know the parent
>> checkout exists in their profiles: when the profiles are initialised the
>> parent checkout may not exist.
>>
>> There may be a better way to do this but it is all done in an antrun
>> plugin. Ant has limited if-else capability in the antrun plugin as it
>> only allows 1 <target> tag per execution. You require multiple <target>
>> tags in the same ant build to have conditional dependencies between
>> them, i.e. check for parent folder checkout and copy, otherwise do a svn
>> checkout.
>>
>> I am going to try moving all this to a build.xml file and call that. It
>> should allow more powerful use of ant. It also separates the ant config
>> from the maven config.
>>
>> If this works the build.xml for ant to checkout the site recursively
>> could be put into a parent.
>>
>> I am still unclear on why this site checkout is required for all the
>> child modules. It is used in the maven-scm-publish-plugin but that
>> should be used on the top level module during a release process. So a
>> simpler fix is to not checkout the site in all the children.
>>
>> A simple test would be to set the poms to not copy the directory for all
>> the child modules and do a dummy release.
>>
>> It's a work in progress...
>
> I'm lost; what's the purpose?

Removing all these copies of the live site. I think I need to look at the code for the commons-release plugin to understand what this folder it used for. It seems to me that it is not used for general development.

>
> Gilles
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email] <mailto:[hidden email]>
> For additional commands, e-mail: [hidden email] <mailto:[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: [RNG] Site layout for modular components (Was: Release [...] RNG [...])

Gilles Sadowski-2
2019-11-14 1:35 UTC+01:00, Alex Herbert <[hidden email]>:

>
>
>> On 13 Nov 2019, at 23:53, Gilles Sadowski <[hidden email]> wrote:
>>
>> Hello.
>>
>> Le mer. 13 nov. 2019 à 18:29, Alex Herbert <[hidden email]
>> <mailto:[hidden email]>> a écrit :
>>>
>>>
>>> On 13/11/2019 13:09, Gilles Sadowski wrote:
>>>> Hi.
>>>>
>>>> Do you think that it would be feasible to have all those fixes applied
>>>> to other modular components (that were based on the [RNG] layout)
>>>> through a common layer (another POM) between those components
>>>> and "commons-parent"?
>>>
>>> Most of the fixes have been in the module site descriptors or items that
>>> should be moved up to commons parent. It may be possible to put the site
>>> descriptors into a parent. IIUC the site descriptors are merged together
>>> before the maven site plugin creates the site. So the fix for the top
>>> right banner could be moved into a common parent. Each child module
>>> would also want to enumerate the past releases of the javadocs. Thus
>>> they would require a site descriptor anyway and the banner fix would
>>> only save 5 lines per site.xml for the banner.
>>
>> Well, I was thinking of whether a multi-layered POM could be more
>> flexible and robust in handling the different type of components (e.g.
>> "multi-module" or not).
>
> I think this would take a bit more reading on exactly how Maven thinks a
> multi-module project should work...

Probably. :-/

>>
>>> I did a big change to the use of svn to checkout the current site. This
>>> was required as the archived javadocs are not in a top level directory.
>>> So each module has to be checked out, the archived javadocs set to be
>>> excluded and then the rest of the site can be checked out.
>>
>> Sorry, I don't follow.
>> Didn't links to the Javadoc of previous versions exist prior to those
>> changes?
>
> Yes. Take a look at the pom.xml on line 464 for prepare-checkout.
>
> This entire section was and is a mystery to me. I don’t know why it is
> required to create a copy of the site locally.

Oh, I seem to remember now that I was hit by this nonsense of
svn checking out the web site when the "site-content" did not
exist!

> The previous code in this maven target was created on the assumption that it
> was a single module project and the archived javadocs for every previous
> version were in a javadocs directory under the top level directory. The same
> code is present in lang, io, text, etc.
>
> The way it works for those projects is incorrect for a multi-module site as
> the archived javadocs are in a different place. It also is a target run in
> every module and so for a full site build with the examples modules you
> ended up with 14 full checkout copies of the RNG site, including the old
> archived javadocs.

Not sure I get what you mean; but I don't think that anything related
to svn should be necessary in the POM file, excerpt perhaps to automate
the creation of an *empty* "site-content" directory (in every module)
in order to prevent downloading the web site.

>
> Anyway I fixed it to work as it should. It checks out the site and ignores
> the old javadocs. I added a fix so child modules copy the parent site
> checkout. This only works if invoked in two stages on a clean git checkout:
>
> mvn -N pre-site && mvn pre-site
>
> This in itself is annoying as you cannot do:
>
> git clone …
> cd commons-rng && mvn site
>
> Without the checkout of the site in each module. I think I can fix this by
> using an external ant build.xml file where you can do if-else logic. But I
> was wondering if I even had to. Perhaps the goal only needs to run in the
> parent, or the dist-archive module for a release.

Not even there (IIUC whet you mean).
What I do is something along the lines:
 $ mvn site site:stage
 $ cd site-content
 $ rm -rf *
 $ cd target/staging
 $ cp -r . ../../site-content
Then
 $ cd ../../site-content
 $ svn add ... the new files (shown with "?" by "svn status")
 $ svn del ... the old files (shown with "!" by "svn status")
 $ svn commit

>
> What I would like to know is:
>
> 1. Do the child modules need this?

I don't think so.

> 2. What is it used for? If it is only for a release then it should be in the
> release profile. Or maybe the commons-release plugin. Using ant to do this
> external execution is just poor (I spent a while fighting ant and it is not
> a programming language, so not suitable for the decisions required for the
> multi-module recursive processing).

I'd favour dropping anything which is not working properly. ;-)

> 3. The top level checkout is used in the release process for manually
> updating the site. But why all the others?

Indeed.
I just created (locally) empty "site-content" and never put anything in
them.  [Would be nice to automate; IIRC, Eric too had some mishaps
with "site-content"...]

>
> If I go to for example commons-rng-client-api and empty the site-content
> folder ‘mvn clean package site’ still creates a site that looks fine.

Sure.
As you state above, why all the svn trickery appears in the POM is
a mistery...

>
> Another bug with multi-module builds is that the following reports are in
> each child module and are duplicates of that in the top level page:
>
> Project information
> - Team
> - SCM
> - Issue Management
> - Mailing lists
> - CI management
> - Distribution management

That's probably because, lacking sufficient knowledge, I copied everything
to each module (using the non-modular Commons Math as a template).
OTOH, I don't think that having the above in each "sub-site" is bad.

> Project reports
> - jira
>
> The ones in the project information menu are harmless and fast to create.
> The jira report takes a long time to generate. I think at least this one
> should be disabled in child modules.

+1
[The more so that it refers to all the issues, not only those that pertain
to the module at hand.]

>
> My motivation is to reduce bloat and and speed up the process of a site
> build. It was annoying when doing the release as you use clean checkouts and
> the download of 14 copies of the full site was slow.

Waiting for the upload is already painful enough: All files are transmitted at
each web site update because of some tag, or date, has changed. :-(

>>>
>>> Since this process is repeated for every module it can get very slow. I
>>> changed the site checkout to copy the directory from the parent if it
>>> was present. There was no solution I could find to fix this to run in a
>>> single maven command as profile activation for all modules is done by
>>> the reactor before any work is performed. Thus if the parent does the
>>> checkout there is no way for all the child modules to know the parent
>>> checkout exists in their profiles: when the profiles are initialised the
>>> parent checkout may not exist.
>>>
>>> There may be a better way to do this but it is all done in an antrun
>>> plugin. Ant has limited if-else capability in the antrun plugin as it
>>> only allows 1 <target> tag per execution. You require multiple <target>
>>> tags in the same ant build to have conditional dependencies between
>>> them, i.e. check for parent folder checkout and copy, otherwise do a svn
>>> checkout.
>>>
>>> I am going to try moving all this to a build.xml file and call that. It
>>> should allow more powerful use of ant. It also separates the ant config
>>> from the maven config.
>>>
>>> If this works the build.xml for ant to checkout the site recursively
>>> could be put into a parent.
>>>
>>> I am still unclear on why this site checkout is required for all the
>>> child modules. It is used in the maven-scm-publish-plugin but that
>>> should be used on the top level module during a release process. So a
>>> simpler fix is to not checkout the site in all the children.
>>>
>>> A simple test would be to set the poms to not copy the directory for all
>>> the child modules and do a dummy release.
>>>
>>> It's a work in progress...
>>
>> I'm lost; what's the purpose?
>
> Removing all these copies of the live site. I think I need to look at the
> code for the commons-release plugin to understand what this folder it used
> for. It seems to me that it is not used for general development.

Sure, as mentioned above, an existing but empty "site-content" and
everything works fine.

Regards,
Gilles

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [RNG] Site layout for modular components (Was: Release [...] RNG [...])

Alex Herbert

On 14/11/2019 01:39, Gilles Sadowski wrote:

> 2019-11-14 1:35 UTC+01:00, Alex Herbert <[hidden email]>:
>>
>>> On 13 Nov 2019, at 23:53, Gilles Sadowski <[hidden email]> wrote:
>>>
>>> Hello.
>>>
>>> Le mer. 13 nov. 2019 à 18:29, Alex Herbert <[hidden email]
>>> <mailto:[hidden email]>> a écrit :
>>>>
>>>> On 13/11/2019 13:09, Gilles Sadowski wrote:
>>>>> Hi.
>>>>>
>>>>> Do you think that it would be feasible to have all those fixes applied
>>>>> to other modular components (that were based on the [RNG] layout)
>>>>> through a common layer (another POM) between those components
>>>>> and "commons-parent"?
>>>> Most of the fixes have been in the module site descriptors or items that
>>>> should be moved up to commons parent. It may be possible to put the site
>>>> descriptors into a parent. IIUC the site descriptors are merged together
>>>> before the maven site plugin creates the site. So the fix for the top
>>>> right banner could be moved into a common parent. Each child module
>>>> would also want to enumerate the past releases of the javadocs. Thus
>>>> they would require a site descriptor anyway and the banner fix would
>>>> only save 5 lines per site.xml for the banner.
>>> Well, I was thinking of whether a multi-layered POM could be more
>>> flexible and robust in handling the different type of components (e.g.
>>> "multi-module" or not).
>> I think this would take a bit more reading on exactly how Maven thinks a
>> multi-module project should work...
> Probably. :-/
>
>>>> I did a big change to the use of svn to checkout the current site. This
>>>> was required as the archived javadocs are not in a top level directory.
>>>> So each module has to be checked out, the archived javadocs set to be
>>>> excluded and then the rest of the site can be checked out.
>>> Sorry, I don't follow.
>>> Didn't links to the Javadoc of previous versions exist prior to those
>>> changes?
>> Yes. Take a look at the pom.xml on line 464 for prepare-checkout.
>>
>> This entire section was and is a mystery to me. I don’t know why it is
>> required to create a copy of the site locally.
> Oh, I seem to remember now that I was hit by this nonsense of
> svn checking out the web site when the "site-content" did not
> exist!
>
>> The previous code in this maven target was created on the assumption that it
>> was a single module project and the archived javadocs for every previous
>> version were in a javadocs directory under the top level directory. The same
>> code is present in lang, io, text, etc.
>>
>> The way it works for those projects is incorrect for a multi-module site as
>> the archived javadocs are in a different place. It also is a target run in
>> every module and so for a full site build with the examples modules you
>> ended up with 14 full checkout copies of the RNG site, including the old
>> archived javadocs.
> Not sure I get what you mean; but I don't think that anything related
> to svn should be necessary in the POM file, excerpt perhaps to automate
> the creation of an *empty* "site-content" directory (in every module)
> in order to prevent downloading the web site.
>
>> Anyway I fixed it to work as it should. It checks out the site and ignores
>> the old javadocs. I added a fix so child modules copy the parent site
>> checkout. This only works if invoked in two stages on a clean git checkout:
>>
>> mvn -N pre-site && mvn pre-site
>>
>> This in itself is annoying as you cannot do:
>>
>> git clone …
>> cd commons-rng && mvn site
>>
>> Without the checkout of the site in each module. I think I can fix this by
>> using an external ant build.xml file where you can do if-else logic. But I
>> was wondering if I even had to. Perhaps the goal only needs to run in the
>> parent, or the dist-archive module for a release.
> Not even there (IIUC whet you mean).
> What I do is something along the lines:
>   $ mvn site site:stage
>   $ cd site-content
>   $ rm -rf *
>   $ cd target/staging
>   $ cp -r . ../../site-content
> Then
>   $ cd ../../site-content
>   $ svn add ... the new files (shown with "?" by "svn status")
>   $ svn del ... the old files (shown with "!" by "svn status")
>   $ svn commit
>
>> What I would like to know is:
>>
>> 1. Do the child modules need this?
> I don't think so.
>
>> 2. What is it used for? If it is only for a release then it should be in the
>> release profile. Or maybe the commons-release plugin. Using ant to do this
>> external execution is just poor (I spent a while fighting ant and it is not
>> a programming language, so not suitable for the decisions required for the
>> multi-module recursive processing).
> I'd favour dropping anything which is not working properly. ;-)
>
>> 3. The top level checkout is used in the release process for manually
>> updating the site. But why all the others?
> Indeed.
> I just created (locally) empty "site-content" and never put anything in
> them.  [Would be nice to automate; IIRC, Eric too had some mishaps
> with "site-content"...]
>
>> If I go to for example commons-rng-client-api and empty the site-content
>> folder ‘mvn clean package site’ still creates a site that looks fine.
> Sure.
> As you state above, why all the svn trickery appears in the POM is
> a mistery...

I have updated the rng parent pom to do a selective checkout of the site.

Any child modules will just get an empty directory. The parent still
gets the full checkout.

I do not know if the full checkout is required. It should at least be a
svn versioned directory so that you can use it to copy a locally staged
site back to the live site (as you describe above).

Since I do not fully understand what exactly is required of this
directory I will leave it with the full checkout for now. This can
always be changed by updating the 'prepare-checkout' execution in the
'setup-checkout' profile.

>
>> Another bug with multi-module builds is that the following reports are in
>> each child module and are duplicates of that in the top level page:
>>
>> Project information
>> - Team
>> - SCM
>> - Issue Management
>> - Mailing lists
>> - CI management
>> - Distribution management
> That's probably because, lacking sufficient knowledge, I copied everything
> to each module (using the non-modular Commons Math as a template).
> OTOH, I don't think that having the above in each "sub-site" is bad.
I've left these.

>
>> Project reports
>> - jira
>>
>> The ones in the project information menu are harmless and fast to create.
>> The jira report takes a long time to generate. I think at least this one
>> should be disabled in child modules.
> +1
> [The more so that it refers to all the issues, not only those that pertain
> to the module at hand.]

I've updated the jira report to filter on the component id. This must be
keyworded in the Jira ticket. I will go through Jira and label those
which are missing component labels.

Currently I have set examples to run a jira report using the 'examples'
tag. This could be further sub-divisioned for each sub component of
examples. Currently this is not done in Jira.

>
>> My motivation is to reduce bloat and and speed up the process of a site
>> build. It was annoying when doing the release as you use clean checkouts and
>> the download of 14 copies of the full site was slow.
> Waiting for the upload is already painful enough: All files are transmitted at
> each web site update because of some tag, or date, has changed. :-(
>
>>>> Since this process is repeated for every module it can get very slow. I
>>>> changed the site checkout to copy the directory from the parent if it
>>>> was present. There was no solution I could find to fix this to run in a
>>>> single maven command as profile activation for all modules is done by
>>>> the reactor before any work is performed. Thus if the parent does the
>>>> checkout there is no way for all the child modules to know the parent
>>>> checkout exists in their profiles: when the profiles are initialised the
>>>> parent checkout may not exist.
>>>>
>>>> There may be a better way to do this but it is all done in an antrun
>>>> plugin. Ant has limited if-else capability in the antrun plugin as it
>>>> only allows 1 <target> tag per execution. You require multiple <target>
>>>> tags in the same ant build to have conditional dependencies between
>>>> them, i.e. check for parent folder checkout and copy, otherwise do a svn
>>>> checkout.
>>>>
>>>> I am going to try moving all this to a build.xml file and call that. It
>>>> should allow more powerful use of ant. It also separates the ant config
>>>> from the maven config.
>>>>
>>>> If this works the build.xml for ant to checkout the site recursively
>>>> could be put into a parent.
>>>>
>>>> I am still unclear on why this site checkout is required for all the
>>>> child modules. It is used in the maven-scm-publish-plugin but that
>>>> should be used on the top level module during a release process. So a
>>>> simpler fix is to not checkout the site in all the children.
>>>>
>>>> A simple test would be to set the poms to not copy the directory for all
>>>> the child modules and do a dummy release.
>>>>
>>>> It's a work in progress...
>>> I'm lost; what's the purpose?
>> Removing all these copies of the live site. I think I need to look at the
>> code for the commons-release plugin to understand what this folder it used
>> for. It seems to me that it is not used for general development.
> Sure, as mentioned above, an existing but empty "site-content" and
> everything works fine.

The site build process should now be a bit faster and cleaner.

Alex

>
> Regards,
> Gilles
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [RNG] Site layout for modular components (Was: Release [...] RNG [...])

Gilles Sadowski-2
Hi.

Le jeu. 14 nov. 2019 à 13:12, Alex Herbert <[hidden email]> a écrit :

>
>
> On 14/11/2019 01:39, Gilles Sadowski wrote:
> > 2019-11-14 1:35 UTC+01:00, Alex Herbert <[hidden email]>:
> >>
> >>> On 13 Nov 2019, at 23:53, Gilles Sadowski <[hidden email]> wrote:
> >>>
> >>> Hello.
> >>>
> >>> Le mer. 13 nov. 2019 à 18:29, Alex Herbert <[hidden email]
> >>> <mailto:[hidden email]>> a écrit :
> >>>>
> >>>> On 13/11/2019 13:09, Gilles Sadowski wrote:
> >>>>> Hi.
> >>>>>
> >>>>> Do you think that it would be feasible to have all those fixes applied
> >>>>> to other modular components (that were based on the [RNG] layout)
> >>>>> through a common layer (another POM) between those components
> >>>>> and "commons-parent"?
> >>>> Most of the fixes have been in the module site descriptors or items that
> >>>> should be moved up to commons parent. It may be possible to put the site
> >>>> descriptors into a parent. IIUC the site descriptors are merged together
> >>>> before the maven site plugin creates the site. So the fix for the top
> >>>> right banner could be moved into a common parent. Each child module
> >>>> would also want to enumerate the past releases of the javadocs. Thus
> >>>> they would require a site descriptor anyway and the banner fix would
> >>>> only save 5 lines per site.xml for the banner.
> >>> Well, I was thinking of whether a multi-layered POM could be more
> >>> flexible and robust in handling the different type of components (e.g.
> >>> "multi-module" or not).
> >> I think this would take a bit more reading on exactly how Maven thinks a
> >> multi-module project should work...
> > Probably. :-/
> >
> >>>> I did a big change to the use of svn to checkout the current site. This
> >>>> was required as the archived javadocs are not in a top level directory.
> >>>> So each module has to be checked out, the archived javadocs set to be
> >>>> excluded and then the rest of the site can be checked out.
> >>> Sorry, I don't follow.
> >>> Didn't links to the Javadoc of previous versions exist prior to those
> >>> changes?
> >> Yes. Take a look at the pom.xml on line 464 for prepare-checkout.
> >>
> >> This entire section was and is a mystery to me. I don’t know why it is
> >> required to create a copy of the site locally.
> > Oh, I seem to remember now that I was hit by this nonsense of
> > svn checking out the web site when the "site-content" did not
> > exist!
> >
> >> The previous code in this maven target was created on the assumption that it
> >> was a single module project and the archived javadocs for every previous
> >> version were in a javadocs directory under the top level directory. The same
> >> code is present in lang, io, text, etc.
> >>
> >> The way it works for those projects is incorrect for a multi-module site as
> >> the archived javadocs are in a different place. It also is a target run in
> >> every module and so for a full site build with the examples modules you
> >> ended up with 14 full checkout copies of the RNG site, including the old
> >> archived javadocs.
> > Not sure I get what you mean; but I don't think that anything related
> > to svn should be necessary in the POM file, excerpt perhaps to automate
> > the creation of an *empty* "site-content" directory (in every module)
> > in order to prevent downloading the web site.
> >
> >> Anyway I fixed it to work as it should. It checks out the site and ignores
> >> the old javadocs. I added a fix so child modules copy the parent site
> >> checkout. This only works if invoked in two stages on a clean git checkout:
> >>
> >> mvn -N pre-site && mvn pre-site
> >>
> >> This in itself is annoying as you cannot do:
> >>
> >> git clone …
> >> cd commons-rng && mvn site
> >>
> >> Without the checkout of the site in each module. I think I can fix this by
> >> using an external ant build.xml file where you can do if-else logic. But I
> >> was wondering if I even had to. Perhaps the goal only needs to run in the
> >> parent, or the dist-archive module for a release.
> > Not even there (IIUC whet you mean).
> > What I do is something along the lines:
> >   $ mvn site site:stage
> >   $ cd site-content
> >   $ rm -rf *
> >   $ cd target/staging
> >   $ cp -r . ../../site-content
> > Then
> >   $ cd ../../site-content
> >   $ svn add ... the new files (shown with "?" by "svn status")
> >   $ svn del ... the old files (shown with "!" by "svn status")
> >   $ svn commit
> >
> >> What I would like to know is:
> >>
> >> 1. Do the child modules need this?
> > I don't think so.
> >
> >> 2. What is it used for? If it is only for a release then it should be in the
> >> release profile. Or maybe the commons-release plugin. Using ant to do this
> >> external execution is just poor (I spent a while fighting ant and it is not
> >> a programming language, so not suitable for the decisions required for the
> >> multi-module recursive processing).
> > I'd favour dropping anything which is not working properly. ;-)
> >
> >> 3. The top level checkout is used in the release process for manually
> >> updating the site. But why all the others?
> > Indeed.
> > I just created (locally) empty "site-content" and never put anything in
> > them.  [Would be nice to automate; IIRC, Eric too had some mishaps
> > with "site-content"...]
> >
> >> If I go to for example commons-rng-client-api and empty the site-content
> >> folder ‘mvn clean package site’ still creates a site that looks fine.
> > Sure.
> > As you state above, why all the svn trickery appears in the POM is
> > a mistery...
>
> I have updated the rng parent pom to do a selective checkout of the site.
>
> Any child modules will just get an empty directory. The parent still
> gets the full checkout.
>
> I do not know if the full checkout is required.

IMO, it is not.

> It should at least be a
> svn versioned directory so that you can use it to copy a locally staged
> site back to the live site (as you describe above).

Sure, but I think that at setup (usually just after time the maven project
is "git clone"d), "site-content" in the top-level directory ("trunk") should
be initialized as "subversion" directory (using the info from the POM, I
guess) but without downloading the contents of the web site since a
developer will hardly ever need it (at least not until wanting to update
the live site).

> Since I do not fully understand what exactly is required of this
> directory I will leave it with the full checkout for now. This can
> always be changed by updating the 'prepare-checkout' execution in the
> 'setup-checkout' profile.

Does this behave as I propose above?

Also: I don't understand why the POM contains this sentence
  "This includes the legacy javadocs for commons-rng-examples release 1.0."

> >
> >> Another bug with multi-module builds is that the following reports are in
> >> each child module and are duplicates of that in the top level page:
> >>
> >> Project information
> >> - Team
> >> - SCM
> >> - Issue Management
> >> - Mailing lists
> >> - CI management
> >> - Distribution management
> > That's probably because, lacking sufficient knowledge, I copied everything
> > to each module (using the non-modular Commons Math as a template).
> > OTOH, I don't think that having the above in each "sub-site" is bad.
> I've left these.
> >
> >> Project reports
> >> - jira
> >>
> >> The ones in the project information menu are harmless and fast to create.
> >> The jira report takes a long time to generate. I think at least this one
> >> should be disabled in child modules.
> > +1
> > [The more so that it refers to all the issues, not only those that pertain
> > to the module at hand.]
>
> I've updated the jira report to filter on the component id. This must be
> keyworded in the Jira ticket. I will go through Jira and label those
> which are missing component labels.

Do you mean "component" or "module"?

If the former, I don't understand.
If the latter, I had rather suggested that the JIRA report is not split into
a separate list for each module; rationale being more or less that issues
management are is a project level task (directly impacting e.g. a release).
Similarly, the "RAT" and "Changes" reports should probably only be visible
at the top-level:
   http://commons.apache.org/proper/commons-rng/project-reports.html
while the "Checkstyle" one should only be visible in a module's "sub-site".

> Currently I have set examples to run a jira report using the 'examples'
> tag. This could be further sub-divisioned for each sub component of
> examples. Currently this is not done in Jira.
>
> >
> >> My motivation is to reduce bloat and and speed up the process of a site
> >> build. It was annoying when doing the release as you use clean checkouts and
> >> the download of 14 copies of the full site was slow.
> > Waiting for the upload is already painful enough: All files are transmitted at
> > each web site update because of some tag, or date, has changed. :-(
> >
> >>>> Since this process is repeated for every module it can get very slow. I
> >>>> changed the site checkout to copy the directory from the parent if it
> >>>> was present. There was no solution I could find to fix this to run in a
> >>>> single maven command as profile activation for all modules is done by
> >>>> the reactor before any work is performed. Thus if the parent does the
> >>>> checkout there is no way for all the child modules to know the parent
> >>>> checkout exists in their profiles: when the profiles are initialised the
> >>>> parent checkout may not exist.
> >>>>
> >>>> There may be a better way to do this but it is all done in an antrun
> >>>> plugin. Ant has limited if-else capability in the antrun plugin as it
> >>>> only allows 1 <target> tag per execution. You require multiple <target>
> >>>> tags in the same ant build to have conditional dependencies between
> >>>> them, i.e. check for parent folder checkout and copy, otherwise do a svn
> >>>> checkout.
> >>>>
> >>>> I am going to try moving all this to a build.xml file and call that. It
> >>>> should allow more powerful use of ant. It also separates the ant config
> >>>> from the maven config.
> >>>>
> >>>> If this works the build.xml for ant to checkout the site recursively
> >>>> could be put into a parent.
> >>>>
> >>>> I am still unclear on why this site checkout is required for all the
> >>>> child modules. It is used in the maven-scm-publish-plugin but that
> >>>> should be used on the top level module during a release process. So a
> >>>> simpler fix is to not checkout the site in all the children.
> >>>>
> >>>> A simple test would be to set the poms to not copy the directory for all
> >>>> the child modules and do a dummy release.
> >>>>
> >>>> It's a work in progress...
> >>> I'm lost; what's the purpose?
> >> Removing all these copies of the live site. I think I need to look at the
> >> code for the commons-release plugin to understand what this folder it used
> >> for. It seems to me that it is not used for general development.
> > Sure, as mentioned above, an existing but empty "site-content" and
> > everything works fine.
>
> The site build process should now be a bit faster and cleaner.
>

Thanks,
Gilles

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

12