[VFS] Could the HDFS connector work with MapR systems?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[VFS] Could the HDFS connector work with MapR systems?

Roger Whitcomb
Hi all,
                I'm using Commons VFS 2.1 along with the HDFS connector and everything is working brilliantly ... until, that is, trying to access HDFS on a MapR system.  Then I get things like null host name exceptions and the like.  So, anyone have any idea whether the HDFS connector even COULD work against MapR?  And if not, anyone tried to build a MapR connector to Commons VFS 2.1?

Thanks in advance,
~Roger Whitcomb
Reply | Threaded
Open this post in threaded view
|

RE: [VFS] Could the HDFS connector work with MapR systems?

garydgregory
I've not seen VFS used in this configuration before. You could try updating to the current HDFS jar files.
Gary 

-------- Original message --------
From: Roger Whitcomb <[hidden email]>
Date: 05/26/2015  10:39  (GMT-08:00)
To: Commons Users List <[hidden email]>
Subject: [VFS] Could the HDFS connector work with MapR systems?

Hi all,
                I'm using Commons VFS 2.1 along with the HDFS connector and everything is working brilliantly ... until, that is, trying to access HDFS on a MapR system.  Then I get things like null host name exceptions and the like.  So, anyone have any idea whether the HDFS connector even COULD work against MapR?  And if not, anyone tried to build a MapR connector to Commons VFS 2.1?

Thanks in advance,
~Roger Whitcomb
Reply | Threaded
Open this post in threaded view
|

RE: [VFS] Could the HDFS connector work with MapR systems?

Roger Whitcomb
So, I'm pretty sure (now) that it DOESN'T work.  There are comments here:  http://answers.mapr.com/questions/9224/apache-hadoop-client-connecting-to-namenode.html and here:  http://doc.mapr.com/display/MapR/Accessing+MapR-FS+in+Java+Applications plus the connector here:  https://github.com/pentaho/pentaho-hdfs-vfs 
that all point to special MapR code being needed, and the standard HDFS client protocols not working.  Plus all my testing seemed to confirm this.

We have figured out a workaround for our configuration, though (we're basically requiring an NFS mount of the MapR file system, so we can just browse the MapR files as native).

Thanks,
~Roger

-----Original Message-----
From: Gary Gregory [mailto:[hidden email]]
Sent: Tuesday, May 26, 2015 3:48 PM
To: Commons Users List
Subject: RE: [VFS] Could the HDFS connector work with MapR systems?

I've not seen VFS used in this configuration before. You could try updating to the current HDFS jar files.
Gary 

-------- Original message --------
From: Roger Whitcomb <[hidden email]>
Date: 05/26/2015  10:39  (GMT-08:00)
To: Commons Users List <[hidden email]>
Subject: [VFS] Could the HDFS connector work with MapR systems?

Hi all,
                I'm using Commons VFS 2.1 along with the HDFS connector and everything is working brilliantly ... until, that is, trying to access HDFS on a MapR system.  Then I get things like null host name exceptions and the like.  So, anyone have any idea whether the HDFS connector even COULD work against MapR?  And if not, anyone tried to build a MapR connector to Commons VFS 2.1?

Thanks in advance,
~Roger Whitcomb

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [VFS] Could the HDFS connector work with MapR systems?

Ted Dunning
The NFS mount trick allows all of the files to appear via the local://*
file system.

Otherwise, it is true that the wire protocol for MapR FS is different from
HDFS.  In fact, the wire protocol for different versions of HDFS are
different as well.

But what is standard is how different implementations of a Hadoop
compatible file system are plugged into the file space.  That should work
for any reasonable application if you have the right jars in your search
path.

One possibility where this would break would be if the client code assumes,
for instance, that all file paths must have a server address.  Otherwise,
well-behaved HDFS client applications should work.  The null host name
exception sounds like there is a format exception going on.

Can I see the some test code and stack traces somewhere?






On Tue, May 26, 2015 at 4:32 PM, Roger Whitcomb <[hidden email]>
wrote:

> So, I'm pretty sure (now) that it DOESN'T work.  There are comments here:
> http://answers.mapr.com/questions/9224/apache-hadoop-client-connecting-to-namenode.html
> and here:
> http://doc.mapr.com/display/MapR/Accessing+MapR-FS+in+Java+Applications
> plus the connector here:  https://github.com/pentaho/pentaho-hdfs-vfs
> that all point to special MapR code being needed, and the standard HDFS
> client protocols not working.  Plus all my testing seemed to confirm this.
>
> We have figured out a workaround for our configuration, though (we're
> basically requiring an NFS mount of the MapR file system, so we can just
> browse the MapR files as native).
>
> Thanks,
> ~Roger
>
> -----Original Message-----
> From: Gary Gregory [mailto:[hidden email]]
> Sent: Tuesday, May 26, 2015 3:48 PM
> To: Commons Users List
> Subject: RE: [VFS] Could the HDFS connector work with MapR systems?
>
> I've not seen VFS used in this configuration before. You could try
> updating to the current HDFS jar files.
> Gary
>
> -------- Original message --------
> From: Roger Whitcomb <[hidden email]>
> Date: 05/26/2015  10:39  (GMT-08:00)
> To: Commons Users List <[hidden email]>
> Subject: [VFS] Could the HDFS connector work with MapR systems?
>
> Hi all,
>                 I'm using Commons VFS 2.1 along with the HDFS connector
> and everything is working brilliantly ... until, that is, trying to access
> HDFS on a MapR system.  Then I get things like null host name exceptions
> and the like.  So, anyone have any idea whether the HDFS connector even
> COULD work against MapR?  And if not, anyone tried to build a MapR
> connector to Commons VFS 2.1?
>
> Thanks in advance,
> ~Roger Whitcomb
>
Reply | Threaded
Open this post in threaded view
|

RE: [VFS] Could the HDFS connector work with MapR systems?

Roger Whitcomb
Hi Ted,
        At this point I gave up and implemented the workaround to browse the NFS mount area instead.  The test case was kind of complicated anyway, since it involves the Pivot VFS File Browser as we as the Commons VFS code.

Thanks,
~Roger

-----Original Message-----
From: Ted Dunning [mailto:[hidden email]]
Sent: Tuesday, May 26, 2015 8:22 PM
To: Commons Users List
Subject: Re: [VFS] Could the HDFS connector work with MapR systems?

The NFS mount trick allows all of the files to appear via the local://* file system.

Otherwise, it is true that the wire protocol for MapR FS is different from HDFS.  In fact, the wire protocol for different versions of HDFS are different as well.

But what is standard is how different implementations of a Hadoop compatible file system are plugged into the file space.  That should work for any reasonable application if you have the right jars in your search path.

One possibility where this would break would be if the client code assumes, for instance, that all file paths must have a server address.  Otherwise, well-behaved HDFS client applications should work.  The null host name exception sounds like there is a format exception going on.

Can I see the some test code and stack traces somewhere?






On Tue, May 26, 2015 at 4:32 PM, Roger Whitcomb <[hidden email]>
wrote:

> So, I'm pretty sure (now) that it DOESN'T work.  There are comments here:
> http://answers.mapr.com/questions/9224/apache-hadoop-client-connecting
> -to-namenode.html
> and here:
> http://doc.mapr.com/display/MapR/Accessing+MapR-FS+in+Java+Application
> s plus the connector here:  
> https://github.com/pentaho/pentaho-hdfs-vfs
> that all point to special MapR code being needed, and the standard
> HDFS client protocols not working.  Plus all my testing seemed to confirm this.
>
> We have figured out a workaround for our configuration, though (we're
> basically requiring an NFS mount of the MapR file system, so we can
> just browse the MapR files as native).
>
> Thanks,
> ~Roger
>
> -----Original Message-----
> From: Gary Gregory [mailto:[hidden email]]
> Sent: Tuesday, May 26, 2015 3:48 PM
> To: Commons Users List
> Subject: RE: [VFS] Could the HDFS connector work with MapR systems?
>
> I've not seen VFS used in this configuration before. You could try
> updating to the current HDFS jar files.
> Gary
>
> -------- Original message --------
> From: Roger Whitcomb <[hidden email]>
> Date: 05/26/2015  10:39  (GMT-08:00)
> To: Commons Users List <[hidden email]>
> Subject: [VFS] Could the HDFS connector work with MapR systems?
>
> Hi all,
>                 I'm using Commons VFS 2.1 along with the HDFS
> connector and everything is working brilliantly ... until, that is,
> trying to access HDFS on a MapR system.  Then I get things like null
> host name exceptions and the like.  So, anyone have any idea whether
> the HDFS connector even COULD work against MapR?  And if not, anyone
> tried to build a MapR connector to Commons VFS 2.1?
>
> Thanks in advance,
> ~Roger Whitcomb
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]