last day (17 days later) » 

3:10 PM
0
Q: Apache-Spark load files from HDFS

Ruofan KongI've written a simple python code sum.py to load directory data from HDFS, and add all first-column numbers for each csv file in directory data. The code shows as follow: import os, sys, inspect, csv ### Current directory path. curr_dir = os.path.split(inspect.getfile(inspect.currentframe()))[0...

 
did you check for you user permissions?
 
Yeah, I make all files in HDFS be public using hdfs $ hdfs dfs -chmod -R 777 /data
 
A bit brutal. Are you still working on EC2?
 
As you may see, this time I worked on my local computer. I want to make sure everything works in my local computer, and then I can move to EC2 HDFS side.
 
if you are working on local computer, you don't need to be reading from hdfs. And also if you are considering EC2. (I used EMR as a matter of fact) You'd better set your data source in s3 and use hdfs if needed to intermediate results.
 
3:10 PM
Thanks for your comment! I just want to see if my local HDFS works with my Spark. Do you know any possible reasons generating the error?
 
Hey!
 
hey!
eliasah
Thanks for your comment on my question!
Actually, I followed the steps below:
Step 1. Format the HDFS:
$ hdfs namenode -format
Step 2. Start namenode and datanode
$ start-dfs.sh
Step 3. Copy the data directory from S3
$ hadoop distcp -D fs.s3n.awsAccessKeyId=... -D fs.s3n.awsSecretAccessKey=... s3n://synapsify-ruofan/data/ hdfs:///
Step 4. Set the data directory permission
$ hdfs dfs -chmod -R 777 /data
Step 5. Check if the data is in HDFS
$ hdfs dfs -ls /data
It shows up the following output:
Found 2 items
-rw-r--r-- 1 ying supergroup 6 2015-07-23 15:46 /data/test1.csv
-rw-r--r-- 1 ying supergroup 6 2015-07-23 15:46 /data/test2.csv
Step 6. Run my python code which I posted on Staackoverflow:
0
Q: Apache-Spark load files from HDFS

Ruofan KongI've written a simple python code sum.py to load directory data from HDFS, and add all first-column numbers for each csv file in directory data. The code shows as follow: import os, sys, inspect, csv ### Current directory path. curr_dir = os.path.split(inspect.getfile(inspect.currentframe()))[0...

Then I got the error...
I have no ideas about the error... I really appreciate for your help!
 
4:00 PM
your port is incorrect
i think
try 8020
it should be by default if you haven't change it
 
Thanks :) let me try 8020
Thanks :) let me try 8020
Still get an error
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: java.net.ConnectException: Call From ying-Inspiron-7537/127.0.1.1 to localhost:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
I think my port should be 9000
It seems different errors. When I try localhost:9000, I got the following error:
IOError: [Errno 2] No such file or directory: 'hdfs://localhost:9000/data/test1.csv'
It's very weird...
 

  last day (17 days later) »