[VFS] Further changes to HDFS Provider for alternate configuration support

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[VFS] Further changes to HDFS Provider for alternate configuration support

Roger Whitcomb

Hi all,

                In trying to solve some customer issues we’re having, mainly to do with trying to browse HDFS files when the Name Node is configured for High-Availability, I found I needed to make some more changes/additions to the VFS HDFS Provider.  I have attached a diff/patch file.  But, a couple of questions:

·         This is a follow-on (basically) to my earlier patch for VFS-555, should I open a new JIRA, or just reopen the existing one?

·         Since this actually changes an API (but which is not released yet), is that an acceptable thing to do?

·         The new properties I have added to HdfsFileSystemConfigBuilder are not actually symmetrical, so they don’t fit the definition of a Java “property”, that is the setters are “setXXX” while the getters are “getXXXs” (plural). The logic behind this is that the setters are called by user code, and each can be called multiple times.  The getters are only (meant to be) called by the HdfsFileSystem object (that is internally to the HdfsProvider), and thus it is a lot easier to get all the settings at once than to have to make multiple calls, with potentially iterators, etc.

·         Is this too big a change to fit in last-minute before the release of VFS 2.1?

 

Patch is attached.

 

Thanks,

~Roger Whitcomb



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: [VFS] Further changes to HDFS Provider for alternate configuration support

dlmarion
I don't see the patch. It might be stripped off by the mail server.

 

From: Roger Whitcomb [mailto:[hidden email]]
Sent: Monday, November 16, 2015 5:12 PM
To: Commons Developers List
Subject: [VFS] Further changes to HDFS Provider for alternate configuration
support

 

Hi all,

                In trying to solve some customer issues we're having, mainly
to do with trying to browse HDFS files when the Name Node is configured for
High-Availability, I found I needed to make some more changes/additions to
the VFS HDFS Provider.  I have attached a diff/patch file.  But, a couple of
questions:

.         This is a follow-on (basically) to my earlier patch for VFS-555,
should I open a new JIRA, or just reopen the existing one?

.         Since this actually changes an API (but which is not released
yet), is that an acceptable thing to do?

.         The new properties I have added to HdfsFileSystemConfigBuilder are
not actually symmetrical, so they don't fit the definition of a Java
"property", that is the setters are "setXXX" while the getters are "getXXXs"
(plural). The logic behind this is that the setters are called by user code,
and each can be called multiple times.  The getters are only (meant to be)
called by the HdfsFileSystem object (that is internally to the
HdfsProvider), and thus it is a lot easier to get all the settings at once
than to have to make multiple calls, with potentially iterators, etc.

.         Is this too big a change to fit in last-minute before the release
of VFS 2.1?

 

Patch is attached.

 

Thanks,

~Roger Whitcomb

Reply | Threaded
Open this post in threaded view
|

Re: [VFS] Further changes to HDFS Provider for alternate configuration support

Pascal Schumacher
Yes, the mailing list server does not allow attachments. Please attach
the patch to a JIRA issue.

Thanks!

Am 17.11.2015 um 00:11 schrieb [hidden email]:

> I don't see the patch. It might be stripped off by the mail server.
>
>  
>
> From: Roger Whitcomb [mailto:[hidden email]]
> Sent: Monday, November 16, 2015 5:12 PM
> To: Commons Developers List
> Subject: [VFS] Further changes to HDFS Provider for alternate configuration
> support
>
>  
>
> Hi all,
>
>                  In trying to solve some customer issues we're having, mainly
> to do with trying to browse HDFS files when the Name Node is configured for
> High-Availability, I found I needed to make some more changes/additions to
> the VFS HDFS Provider.  I have attached a diff/patch file.  But, a couple of
> questions:
>
> .         This is a follow-on (basically) to my earlier patch for VFS-555,
> should I open a new JIRA, or just reopen the existing one?
>
> .         Since this actually changes an API (but which is not released
> yet), is that an acceptable thing to do?
>
> .         The new properties I have added to HdfsFileSystemConfigBuilder are
> not actually symmetrical, so they don't fit the definition of a Java
> "property", that is the setters are "setXXX" while the getters are "getXXXs"
> (plural). The logic behind this is that the setters are called by user code,
> and each can be called multiple times.  The getters are only (meant to be)
> called by the HdfsFileSystem object (that is internally to the
> HdfsProvider), and thus it is a lot easier to get all the settings at once
> than to have to make multiple calls, with potentially iterators, etc.
>
> .         Is this too big a change to fit in last-minute before the release
> of VFS 2.1?
>
>  
>
> Patch is attached.
>
>  
>
> Thanks,
>
> ~Roger Whitcomb
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VFS] Further changes to HDFS Provider for alternate configuration support

Roger Whitcomb
Okay, I'll go ahead and make a new JIRA issue, then, and attach the patch there.

Thanks,
~Roger

________________________________________
From: Pascal Schumacher <[hidden email]>
Sent: Tuesday, November 17, 2015 9:48 AM
To: Commons Developers List
Subject: Re: [VFS] Further changes to HDFS Provider for alternate configuration support

Yes, the mailing list server does not allow attachments. Please attach
the patch to a JIRA issue.

Thanks!

Am 17.11.2015 um 00:11 schrieb [hidden email]:

> I don't see the patch. It might be stripped off by the mail server.
>
>
>
> From: Roger Whitcomb [mailto:[hidden email]]
> Sent: Monday, November 16, 2015 5:12 PM
> To: Commons Developers List
> Subject: [VFS] Further changes to HDFS Provider for alternate configuration
> support
>
>
>
> Hi all,
>
>                  In trying to solve some customer issues we're having, mainly
> to do with trying to browse HDFS files when the Name Node is configured for
> High-Availability, I found I needed to make some more changes/additions to
> the VFS HDFS Provider.  I have attached a diff/patch file.  But, a couple of
> questions:
>
> .         This is a follow-on (basically) to my earlier patch for VFS-555,
> should I open a new JIRA, or just reopen the existing one?
>
> .         Since this actually changes an API (but which is not released
> yet), is that an acceptable thing to do?
>
> .         The new properties I have added to HdfsFileSystemConfigBuilder are
> not actually symmetrical, so they don't fit the definition of a Java
> "property", that is the setters are "setXXX" while the getters are "getXXXs"
> (plural). The logic behind this is that the setters are called by user code,
> and each can be called multiple times.  The getters are only (meant to be)
> called by the HdfsFileSystem object (that is internally to the
> HdfsProvider), and thus it is a lot easier to get all the settings at once
> than to have to make multiple calls, with potentially iterators, etc.
>
> .         Is this too big a change to fit in last-minute before the release
> of VFS 2.1?
>
>
>
> Patch is attached.
>
>
>
> Thanks,
>
> ~Roger Whitcomb
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: [VFS] Further changes to HDFS Provider for alternate configuration support

Roger Whitcomb
In reply to this post by Pascal Schumacher
I have created https://issues.apache.org/jira/browse/VFS-586 and attached the proposed patch there for your consideration.

The original questions I had still apply, and will be added to the issue.

Thanks,
~Roger Whitcomb

-----Original Message-----
From: Pascal Schumacher [mailto:[hidden email]]
Sent: Tuesday, November 17, 2015 9:48 AM
To: Commons Developers List <[hidden email]>
Subject: Re: [VFS] Further changes to HDFS Provider for alternate configuration support

Yes, the mailing list server does not allow attachments. Please attach the patch to a JIRA issue.

Thanks!

Am 17.11.2015 um 00:11 schrieb [hidden email]:

> I don't see the patch. It might be stripped off by the mail server.
>
>  
>
> From: Roger Whitcomb [mailto:[hidden email]]
> Sent: Monday, November 16, 2015 5:12 PM
> To: Commons Developers List
> Subject: [VFS] Further changes to HDFS Provider for alternate
> configuration support
>
>  
>
> Hi all,
>
>                  In trying to solve some customer issues we're having,
> mainly to do with trying to browse HDFS files when the Name Node is
> configured for High-Availability, I found I needed to make some more
> changes/additions to the VFS HDFS Provider.  I have attached a
> diff/patch file.  But, a couple of
> questions:
>
> .         This is a follow-on (basically) to my earlier patch for VFS-555,
> should I open a new JIRA, or just reopen the existing one?
>
> .         Since this actually changes an API (but which is not released
> yet), is that an acceptable thing to do?
>
> .         The new properties I have added to HdfsFileSystemConfigBuilder are
> not actually symmetrical, so they don't fit the definition of a Java
> "property", that is the setters are "setXXX" while the getters are "getXXXs"
> (plural). The logic behind this is that the setters are called by user
> code, and each can be called multiple times.  The getters are only
> (meant to be) called by the HdfsFileSystem object (that is internally
> to the HdfsProvider), and thus it is a lot easier to get all the
> settings at once than to have to make multiple calls, with potentially iterators, etc.
>
> .         Is this too big a change to fit in last-minute before the release
> of VFS 2.1?
>
>  
>
> Patch is attached.
>
>  
>
> Thanks,
>
> ~Roger Whitcomb
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: [VFS] Further changes to HDFS Provider for alternate configuration support

Roger Whitcomb
In reply to this post by dlmarion
I have opened issue https://issues.apache.org/jira/browse/VFS-586 with my proposed patch attached, and these questions included in the issue.  Updated the patch with some things I noticed after the initial post also.

Thanks,
~Roger

-----Original Message-----
From: [hidden email] [mailto:[hidden email]]
Sent: Monday, November 16, 2015 3:11 PM
To: 'Commons Developers List' <[hidden email]>
Subject: RE: [VFS] Further changes to HDFS Provider for alternate configuration support

I don't see the patch. It might be stripped off by the mail server.

 

From: Roger Whitcomb [mailto:[hidden email]]
Sent: Monday, November 16, 2015 5:12 PM
To: Commons Developers List
Subject: [VFS] Further changes to HDFS Provider for alternate configuration support

 

Hi all,

                In trying to solve some customer issues we're having, mainly to do with trying to browse HDFS files when the Name Node is configured for High-Availability, I found I needed to make some more changes/additions to the VFS HDFS Provider.  I have attached a diff/patch file.  But, a couple of
questions:

.         This is a follow-on (basically) to my earlier patch for VFS-555,
should I open a new JIRA, or just reopen the existing one?

.         Since this actually changes an API (but which is not released
yet), is that an acceptable thing to do?

.         The new properties I have added to HdfsFileSystemConfigBuilder are
not actually symmetrical, so they don't fit the definition of a Java "property", that is the setters are "setXXX" while the getters are "getXXXs"
(plural). The logic behind this is that the setters are called by user code, and each can be called multiple times.  The getters are only (meant to be) called by the HdfsFileSystem object (that is internally to the HdfsProvider), and thus it is a lot easier to get all the settings at once than to have to make multiple calls, with potentially iterators, etc.

.         Is this too big a change to fit in last-minute before the release
of VFS 2.1?

 

Patch is attached.

 

Thanks,

~Roger Whitcomb


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]