brisk快速测试记录。
参考链接：
http://www.datastax.com/docs/0.8/brisk/about_pig
阅读这篇文章的其余部分 »
本文来自: brisk调试部署全纪录
标签 Tag : hive
FAILED: Error in metadata: javax.jdo.JDOFatalDataStoreException: Cannot get a connection, pool error Could not create a validated object, cause: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.
NestedThrowables:
org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Could not create a validated object, cause: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
Hive history file=/tmp/dev/hive_job_log_dev_201107062337_381665684.txt
FAILED: Error in semantic analysis: line 1:83 Exception while processing raw_daily_stats_table: Unable to fetch table raw_daily_stats_table
查看hive配置文件/etc/hive/conf/hive-default.xml,找到你的元数据存放位置
|
1 2 3 4 5 |
<property> <name>hive.metastore.warehouse.dir</name> <value>/user/hive/warehouse</value> <description>location of default database for the warehouse</description> </property> |
打开hdfs目录发现
/user/hive/warehouse
|
1 2 |
raw_daily_pagecounts_table dir 2011-03-28 15:39 rwxr-xr-x dev supergroup raw_daily_stats_table dir 2011-07-06 23:03 rwxr-xr-x root supergroup |
raw_daily_stats_table 目录的权限成root了，但是我是以dev身份执行的，
执行：
|
1 |
hadoop fs -chown -R dev:dev /user/hive/warehouse/raw_daily_stats_table |
结果发现还是报，神啊
FAILED: Error in metadata: javax.jdo.JDOFatalDataStoreException: Cannot get a connection, pool error Could not create a validated object, cause: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.
NestedThrowables:
org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Could not create a validated object, cause: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
打开配置文件/etc/hive/conf/hive-site.xml发现如下节点
|
1 2 3 4 5 |
<property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:derby:;databaseName=/var/lib/hive/metastore/metastore_db;create=true</value> <description>JDBC connect string for a JDBC metastore</description> </property> |
然后定位到相应目录
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
[dev@platformB metastore]$ ls -al total 16 drwxrwxrwt 4 root root 4096 Mar 28 15:25 . drwxr-xr-x 3 root root 4096 Mar 28 15:22 .. drwxrwxr-x 5 dev dev 4096 Jul 6 23:51 metastore_db drwxrwxrwt 3 root root 4096 Mar 28 15:22 scripts [dev@platformB metastore]$ cd metastore_db/ [dev@platformB metastore_db]$ ls dbex.lck db.lck log seg0 service.properties tmp [dev@platformB metastore_db]$ ls -al total 32 drwxrwxr-x 5 dev dev 4096 Jul 6 23:51 . drwxrwxrwt 4 root root 4096 Mar 28 15:25 .. -rw-rw-r-- 1 dev dev 4 Jul 6 23:03 dbex.lck -rw-r--r-- 1 root root 38 Jul 6 23:03 db.lck drwxrwxr-x 2 dev dev 4096 Jul 6 15:22 log drwxrwxr-x 2 dev dev 4096 Apr 6 13:04 seg0 -rw-rw-r-- 1 dev dev 860 Mar 28 15:25 service.properties drwxrwxr-x 2 dev dev 4096 Jul 6 23:51 tmp [dev@platformB metastore_db]$ |
db.lck 干掉， dbex.lck干掉
再跑hadoop相关脚本，ok~
本文来自: Hive derby lock及目录权限错误
https://github.com/datawrangling/trendingtopics
https://github.com/datawrangling/spatialanalytics
搭建trendingtopics,步骤。
环境准备
|
1 2 3 4 5 |
sudo apt-get install ruby sudo gem install rails -include-dependenciesgem /home/cloudera/.gem/ruby/1.8/bin git clone git://github.com/datawrangling/trendingtopics.git |
配置文件
|
1 2 3 |
cd trendingtopics cp config/config.yml.example config/config.yml cp config/database.yml.example config/database.yml |
安装
|
1 |
rake gems:install |
如果保错：undefined local variable or method `version_requirements’
vi config/environment.rb
在开头加入：
|
1 2 3 4 5 6 7 8 9 10 |
if Gem::VERSION >= "1.3.6" module Rails class GemDependency def requirement r = super (r == Gem::Requirement.default) ? nil : r end end end end |
安装mysql client和mysql gem
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
//解压msyql源码包 ./configure //./configure --prefix=/usr/local/mysql make install //错误error: sys/ttydefaults.h: No such file or directory'： //http://phaseshiftllc.com/archives/2008/10/26/installing-mysql-gem-on-windows-cygwin-for-rails //make distclean //./configure --without-readline CFLAGS=-O2 //./configure --prefix /usr/local/mysql --without-server --without-readline --without-libeditCFLAGS=-O2 CFLAGS=-O2 CXXFLAGS=-O2 //make install //gem install mysql cp support-files/my-medium.cnf /etc/my.cnf cd /usr/local/mysql vi /etc/my.conf [client] 中加入 protocol=TCP //ref:http://www.phpvim.net/os/windows/build-mysql-client-on-cygwin.html mysql -h localhost -u root gem install mysql -include-dependenciesgem mysqld_safe --user=mysql & mysql -h localhost -u root -p |
配置数据库连接
|
1 2 3 4 5 6 |
replace socket: /tmp/mysql.sock with username: root password: 555555 host: localhost |
安装数据库
|
1 2 |
rake db:create rake db:migrate |
生成100条文章来做demo数据
|
1 |
rake db:develop |
|
1 2 3 4 5 6 7 8 9 10 11 |
//启动server script/server //报错，缺少包，执行如下 rake gems:install gem sources -a http://gems.github.com gem install jpignata-bossman //再执行 script/server |
server启动后，访问地址http://localhost:3000/
|
1 2 3 4 5 6 7 8 |
$ script/server => Booting WEBrick => Rails 2.3.2 application starting on http://0.0.0.0:3000 => Call with -d to detach => Ctrl-C to shutdown server [2011-03-24 13:50:11] INFO WEBrick 1.3.1 [2011-03-24 13:50:11] INFO ruby 1.8.7 (2008-08-11) [i386-cygwin] [2011-03-24 13:50:11] INFO WEBrick::HTTPServer#start: pid=4760 port=3000 |
报错：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
242093716 [main] bash 4404 exception::handle: Exception: STATUS_ACCESS_VIOLATION 242095173 [main] bash 4404 open_stackdumpfile: Dumping stack trace to bash.exe.s tackdump 242305333 [main] bash 2116 exception::handle: Exception: STATUS_ACCESS_VIOLATION 242306088 [main] bash 2116 open_stackdumpfile: Dumping stack trace to bash.exe.s tackdump 242617570 [main] bash 4032 exception::handle: Exception: STATUS_ACCESS_VIOLATION 242619190 [main] bash 4032 open_stackdumpfile: Dumping stack trace to bash.exe.s tackdump 243121910 [main] bash 3596 exception::handle: Exception: STATUS_ACCESS_VIOLATION 243123323 [main] bash 3596 open_stackdumpfile: Dumping stack trace to bash.exe.s tackdump 243458891 [main] bash 4968 fork: child -1 - died waiting for longjmp before init ialization, retry 0, exit code 0x600, errno 11 bash: fork: Resource temporarily unavailable |
|
1 2 3 4 5 6 7 |
//原因：temp放在虚拟磁盘，cygwin访问的权限不够 //如果是其他的原因可尝试如下方法： rebaseall: only ash processes are allowed during rebasing Exit all Cygwin processes and stop all Cygwin services. Execute ash from Start/Run... or a cmd or command window. Execute '/bin/rebaseall' from ash. from:http://cygwin.com/ml/cygwin/2005-09/msg00919.html |
创建表 CREATE TABLE raw_daily_stats_table1 (redirect_title STRING, dates STRING, pageviews STRING, total_pageviews BIGINT, monthly_trend DOUBLE) ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘\t’ STORED AS TEXTFILE; 加载数据 LOAD DATA INPATH ‘/home/dev/finalresult-a’ INTO TABLE raw_daily_stats_table; //文件路径为hadoop的文件路径，上面的路径对应为hdfs://platformB/home/dev/finalresult-a
|
1 2 3 4 |
hive> LOAD DATA INPATH '/home/dev/finalresult-a' INTO TABLE raw_daily_stats_table; Loading data to table raw_daily_stats_table OK Time taken: 4.927 seconds |
加载的时候如果报加载失败，检查你的hdfs，会发现生成了一个你的文件名+_copy_1的文件，然后你load这个文件就成了。 hive> show tables > ; FAILED: Error in metadata: javax.jdo.JDOFatalDataStoreException: Failed to start database ‘/var/lib/hive/metastore/metastore_db’, see the next exception for details. NestedThrowables: java.sql.SQLException: Failed to start database ‘/var/lib/hive/metastore/metastore_db’, se e the next exception for details. FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask hive> cat derby.log ============= begin nested exception, level (3) =========== ERROR XSDB6: Another instance of Derby may have already booted the database /var/lib/hive/ metastore/metastore_db. at org.apache.derby.iapi.error.StandardException.newException(Unknown Source) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.privGetJBMSLockOnDB(Un known Source) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.getJBMSLockOnDB(Unknow n Source) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.boot(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source) at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source) at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source ) 原来异常退出造成前面的访问derby进程还在，而derby是文件型的存储,每次只能一个进程打开，so，你懂的,看来生成环境使用mysql才是王道，打开配置文件hive-default.xml
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
hive.metastore.warehouse.dir /user/hive_remote/warehouse hive.metastore.local true javax.jdo.option.ConnectionURL jdbc:mysql://localhost/hive_remote?createDatabaseIfNotExist=true javax.jdo.option.ConnectionDriverName com.mysql.jdbc.Driver javax.jdo.option.ConnectionUserName root javax.jdo.option.ConnectionPassword dandan |
hive查询及排序： select * from raw_daily_stats_table sort by monthly_trend; select * from raw_daily_stats_table sort by monthly_trend desc limit 10; http://www.fuzhijie.me/?p=377 http://wiki.apache.org/hadoop/Hive/AdminManual/MetastoreAdmin
本文来自: 搭建trendingtopics