Wondering VFS will support Read/Write on Google Cloud Storage?

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Wondering VFS will support Read/Write on Google Cloud Storage?

楊閔富
Hi

I am using Zeppelin on Google Cloud DataProc and want make the notebooks to
be stored on the Google Cloud Storage.
At first, I used `gcsfuse`(https://github.com/GoogleCloudPlatform/gcsfuse)
to mount my Google Cloud Storage bucket as a local directory, but VFS used
by Zeppelin to manage files said the directory didn't exist.

So I am wondering is support for Google Cloud Storage is in the VFS plan?
If yes, It's my pleasure to implement it. If not, why?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Wondering VFS will support Read/Write on Google Cloud Storage?

楊閔富
Another question is that VFS cannot identify directory mounted by fuse?

2017-06-15 8:47 GMT+08:00 楊閔富 <[hidden email]>:

> Hi
>
> I am using Zeppelin on Google Cloud DataProc and want make the notebooks
> to be stored on the Google Cloud Storage.
> At first, I used `gcsfuse`(https://github.com/GoogleCloudPlatform/gcsfuse)
> to mount my Google Cloud Storage bucket as a local directory, but VFS used
> by Zeppelin to manage files said the directory didn't exist.
>
> So I am wondering is support for Google Cloud Storage is in the VFS plan?
> If yes, It's my pleasure to implement it. If not, why?
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Wondering VFS will support Read/Write on Google Cloud Storage?

Bernd Eckenfels
Hello,

The Local file provider should be able to deal with fuse mounted directories. It is hard to say what the problem is. Can you tell us how the path is named, what operations you tried, what exceptions are happening and what `ls` is returning.



Gruss
Bernd
--
http://bernd.eckenfels.net
________________________________
From: 楊閔富 <[hidden email]>
Sent: Thursday, June 15, 2017 3:03:12 AM
To: [hidden email]
Subject: Re: Wondering VFS will support Read/Write on Google Cloud Storage?

Another question is that VFS cannot identify directory mounted by fuse?

2017-06-15 8:47 GMT+08:00 楊閔富 <[hidden email]>:

> Hi
>
> I am using Zeppelin on Google Cloud DataProc and want make the notebooks
> to be stored on the Google Cloud Storage.
> At first, I used `gcsfuse`(https://github.com/GoogleCloudPlatform/gcsfuse)
> to mount my Google Cloud Storage bucket as a local directory, but VFS used
> by Zeppelin to manage files said the directory didn't exist.
>
> So I am wondering is support for Google Cloud Storage is in the VFS plan?
> If yes, It's my pleasure to implement it. If not, why?
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Wondering VFS will support Read/Write on Google Cloud Storage?

楊閔富
At first I mount my Google Cloud Storage onto directory /tmp/gcs using
`gcsfuse`,
then I create a small project: https://github.com/tilumi/apache-vfs-debug
for testing it VFS can detect my mounting point or not.

the execution log:
```
$ls -al /tmp/gcs
total 0
drwxrwxrwx 1 root root 0 Jun 17 00:42 hive-warehouse
drwxrwxrwx 1 root root 0 Jun 17 00:42 softwares
drwxrwxrwx 1 root root 0 Jun 17 00:42 zeppelin-notebooks
$sudo java -cp apache-vfs-debug-1.0-SNAPSHOT-jar-with-dependencies.jar
Sample "file:///tmp/gcs"
false
$sudo java -cp apache-vfs-debug-1.0-SNAPSHOT-jar-with-dependencies.jar
Sample "file:///tmp"
true
```
It seems that it cannot correctly identify the directory mounted by gcsfuse.


2017-06-15 16:48 GMT+08:00 Bernd Eckenfels <[hidden email]>:

> Hello,
>
> The Local file provider should be able to deal with fuse mounted
> directories. It is hard to say what the problem is. Can you tell us how the
> path is named, what operations you tried, what exceptions are happening and
> what `ls` is returning.
>
>
>
> Gruss
> Bernd
> --
> http://bernd.eckenfels.net
> ________________________________
> From: 楊閔富 <[hidden email]>
> Sent: Thursday, June 15, 2017 3:03:12 AM
> To: [hidden email]
> Subject: Re: Wondering VFS will support Read/Write on Google Cloud Storage?
>
> Another question is that VFS cannot identify directory mounted by fuse?
>
> 2017-06-15 8:47 GMT+08:00 楊閔富 <[hidden email]>:
>
> > Hi
> >
> > I am using Zeppelin on Google Cloud DataProc and want make the notebooks
> > to be stored on the Google Cloud Storage.
> > At first, I used `gcsfuse`(https://github.com/
> GoogleCloudPlatform/gcsfuse)
> > to mount my Google Cloud Storage bucket as a local directory, but VFS
> used
> > by Zeppelin to manage files said the directory didn't exist.
> >
> > So I am wondering is support for Google Cloud Storage is in the VFS plan?
> > If yes, It's my pleasure to implement it. If not, why?
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Wondering VFS will support Read/Write on Google Cloud Storage?

Bernd Eckenfels
In reply to this post by 楊閔富
Thanks for the try.

This is pretty strange since your mount path looks simple and VFS file:
provider is not doing much besides using normal Java API to access the
filesystem. so this looks more like a Java/fuse/gcs-fuse problem.

Can you try

File f = new File("/tmp/gcs"); Systen,out.println("f exists" + f.exists() +
" is dir " + f.isDirectory() + " is file " + f.isFile() + " norm " +
f.getCanonicalPath() + " len " + f.length())

So we can see what the responses are?

Gruss
Bernd
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Wondering VFS will support Read/Write on Google Cloud Storage?

楊閔富
Hi

I've tried and the log is as follows:

```
$ ls -al /tmp/gcs
total 0
drwxrwxrwx 1 root root 0 Jun 19 07:37 hive-warehouse
drwxrwxrwx 1 root root 0 Jun 19 07:37 softwares
drwxrwxrwx 1 root root 0 Jun 19 07:37 zeppelin-notebooks
$ sudo java -cp apache-vfs-debug-1.0-SNAPSHOT-jar-with-dependencies.jar
Sample "file:///tmp"
Jun 19, 2017 7:38:20 AM
org.apache.commons.vfs2.impl.StandardFileSystemManager info
INFO: Using "/tmp/vfs_cache" as temporary files store.
true
/tmp exists: true, is dir: true, is file: false, norm: /tmp, len: 4096
$ sudo java -cp apache-vfs-debug-1.0-SNAPSHOT-jar-with-dependencies.jar
Sample "file:///tmp/gcs"
Jun 19, 2017 7:38:26 AM
org.apache.commons.vfs2.impl.StandardFileSystemManager info
INFO: Using "/tmp/vfs_cache" as temporary files store.
false
/tmp/gcs exists: false, is dir: false, is file: false, norm: /tmp/gcs, len:
0
```

2017-06-19 3:43 GMT+08:00 Bernd <[hidden email]>:

> Thanks for the try.
>
> This is pretty strange since your mount path looks simple and VFS file:
> provider is not doing much besides using normal Java API to access the
> filesystem. so this looks more like a Java/fuse/gcs-fuse problem.
>
> Can you try
>
> File f = new File("/tmp/gcs"); Systen,out.println("f exists" + f.exists() +
> " is dir " + f.isDirectory() + " is file " + f.isFile() + " norm " +
> f.getCanonicalPath() + " len " + f.length())
>
> So we can see what the responses are?
>
> Gruss
> Bernd
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Wondering VFS will support Read/Write on Google Cloud Storage?

楊閔富
I think the gcsfuse mounted directory cannot even be identified by Java,
I'll report this issue to `gcsfuse` community.

2017-06-19 15:39 GMT+08:00 楊閔富 <[hidden email]>:

> Hi
>
> I've tried and the log is as follows:
>
> ```
> $ ls -al /tmp/gcs
> total 0
> drwxrwxrwx 1 root root 0 Jun 19 07:37 hive-warehouse
> drwxrwxrwx 1 root root 0 Jun 19 07:37 softwares
> drwxrwxrwx 1 root root 0 Jun 19 07:37 zeppelin-notebooks
> $ sudo java -cp apache-vfs-debug-1.0-SNAPSHOT-jar-with-dependencies.jar
> Sample "file:///tmp"
> Jun 19, 2017 7:38:20 AM org.apache.commons.vfs2.impl.StandardFileSystemManager
> info
> INFO: Using "/tmp/vfs_cache" as temporary files store.
> true
> /tmp exists: true, is dir: true, is file: false, norm: /tmp, len: 4096
> $ sudo java -cp apache-vfs-debug-1.0-SNAPSHOT-jar-with-dependencies.jar
> Sample "file:///tmp/gcs"
> Jun 19, 2017 7:38:26 AM org.apache.commons.vfs2.impl.StandardFileSystemManager
> info
> INFO: Using "/tmp/vfs_cache" as temporary files store.
> false
> /tmp/gcs exists: false, is dir: false, is file: false, norm: /tmp/gcs,
> len: 0
> ```
>
> 2017-06-19 3:43 GMT+08:00 Bernd <[hidden email]>:
>
>> Thanks for the try.
>>
>> This is pretty strange since your mount path looks simple and VFS file:
>> provider is not doing much besides using normal Java API to access the
>> filesystem. so this looks more like a Java/fuse/gcs-fuse problem.
>>
>> Can you try
>>
>> File f = new File("/tmp/gcs"); Systen,out.println("f exists" + f.exists()
>> +
>> " is dir " + f.isDirectory() + " is file " + f.isFile() + " norm " +
>> f.getCanonicalPath() + " len " + f.length())
>>
>> So we can see what the responses are?
>>
>> Gruss
>> Bernd
>>
>
>
Loading...