记录生活
分类 Category : Linux

jekyll&es-doc部署脚本

<Date: 2011-11-19> <Author: medcl> <Category: Linux, VPS维护日志> 发表评论

https://github.com/mojombo/jekyll/wiki/Deployment

jekyll&es-doc部署脚本
阅读这篇文章的其余部分 »

本文来自: jekyll&es-doc部署脚本

, ,

ubuntu在线升级到11.10

<Date: 2011-09-06> <Author: medcl> <Category: Linux> 发表评论 
sudo gedit  /etc/apt/sources.list

贴进去

#deb http://cn.archive.ubuntu.com/ubuntu/ natty main restricted universe multiverse
#deb http://cn.archive.ubuntu.com/ubuntu/ natty-security main restricted universe multiverse
#deb http://cn.archive.ubuntu.com/ubuntu/ natty-updates main restricted universe multiverse
#deb http://cn.archive.ubuntu.com/ubuntu/ natty-proposed main restricted universe multiverse
#deb http://cn.archive.ubuntu.com/ubuntu/ natty-backports main restricted universe multiverse
#deb-src http://cn.archive.ubuntu.com/ubuntu/ natty main restricted universe multiverse
#deb-src http://cn.archive.ubuntu.com/ubuntu/ natty-security main restricted universe multiverse
#deb-src http://cn.archive.ubuntu.com/ubuntu/ natty-updates main restricted universe multiverse
#deb-src http://cn.archive.ubuntu.com/ubuntu/ natty-proposed main restricted universe multiverse
#deb-src http://cn.archive.ubuntu.com/ubuntu/ natty-backports main restricted universe multiverse
 
#deb http://ppa.launchpad.net/wengxt/fcitx-nightly/ubuntu natty main
 
deb http://cn.archive.ubuntu.com/ubuntu/ oneiric main restricted universe multiverse
deb http://cn.archive.ubuntu.com/ubuntu/ oneiric-security main restricted universe multiverse
deb http://cn.archive.ubuntu.com/ubuntu/ oneiric-updates main restricted universe multiverse
deb http://cn.archive.ubuntu.com/ubuntu/ oneiric-proposed main restricted universe multiverse
deb http://cn.archive.ubuntu.com/ubuntu/ oneiric-backports main restricted universe multiverse
deb-src http://cn.archive.ubuntu.com/ubuntu/ oneiric main restricted universe multiverse
deb-src http://cn.archive.ubuntu.com/ubuntu/ oneiric-security main restricted universe multiverse
deb-src http://cn.archive.ubuntu.com/ubuntu/ oneiric-updates main restricted universe multiverse
deb-src http://cn.archive.ubuntu.com/ubuntu/ oneiric-proposed main restricted universe multiverse
deb-src http://cn.archive.ubuntu.com/ubuntu/ oneiric-backports main restricted universe multiverse
sudo apt-get update
sudo apt-get -d dist-upgrade
sudo update-manager -d

在弹出都更新界面中，点“部分升级”，重启，搞定。

本文来自: ubuntu在线升级到11.10

, ,

jps&jstack&jmap

<Date: 2011-07-15> <Author: medcl> <Category: Linux> 1 条评论
  • : List your processes
  • : Get the call stack for a given Java process
  • jmap: Show Memory Map

    • jinfo
    jps -mlvV
    jps -l -m 

[dev@pingtai config]$ jps
2446 Jps
434 
[dev@pingtai config]$ jstack 434|more
2011-08-10 14:59:05
Full thread dump Java HotSpot(TM) 64-Bit Server VM (16.3-b01 mixed mode):

"GC Daemon" daemon prio=10 tid=0x00002aaafc61d000 nid=0x959 in Object.wait() [0x0000000040e
59000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aaab1133718> (a sun.misc.GC$LatencyLock)
	at sun.misc.GC$Daemon.run(GC.java:100)
	- locked <0x00002aaab1133718> (a sun.misc.GC$LatencyLock)

"RMI Reaper" prio=10 tid=0x00002aaafc609000 nid=0x958 in Object.wait() [0x0000000041762000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aaab136fd88> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
	- locked <0x00002aaab136fd88> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
	at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)
	at java.lang.Thread.run(Thread.java:619)

"RMI TCP Accept-9492" daemon prio=10 tid=0x00002aaafc61e000 nid=0x957 runnable [0x000000004
0225000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
	- locked <0x00002aaaaf7f9548> (a java.net.SocksSocketImpl)
	at java.net.ServerSocket.implAccept(ServerSocket.java:453)
	at java.net.ServerSocket.accept(ServerSocket.java:421)
	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.jav
a:369)
	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
	at java.lang.Thread.run(Thread.java:619)
... ...

[dev@pingtai config]$ jmap 434
Attaching to process ID 434, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 16.3-b01
0x0000000040000000	49K	/home/dev/jdk1.6.0_20/bin/java
0x0000003983600000	136K	/lib64/ld-2.5.so
0x0000003983a00000	1677K	/lib64/libc-2.5.so
0x0000003983e00000	22K	/lib64/libdl-2.5.so
0x0000003984200000	600K	/lib64/libm-2.5.so
0x0000003984600000	142K	/lib64/libpthread-2.5.so
0x0000003984e00000	52K	/lib64/librt-2.5.so
0x0000003987a00000	111K	/lib64/libnsl-2.5.so
0x00002aaaaaabf000	65K	/home/dev/jdk1.6.0_20/jre/lib/amd64/libverify.so
0x00002aaaaabce000	229K	/home/dev/jdk1.6.0_20/jre/lib/amd64/libjava.so
0x00002aaaaacff000	54K	/home/dev/jdk1.6.0_20/jre/lib/amd64/native_threads/libhpi.so
0x00002aaaaae1d000	52K	/lib64/libnss_files-2.5.so
0x00002aaaab028000	89K	/home/dev/jdk1.6.0_20/jre/lib/amd64/libzip.so
0x00002aaafb02b000	37K	/home/dev/jdk1.6.0_20/jre/lib/amd64/libmanagement.so
0x00002aaafb132000	110K	/home/dev/jdk1.6.0_20/jre/lib/amd64/libnet.so
0x00002aaafbb7f000	43K	/home/dev/jdk1.6.0_20/jre/lib/amd64/libnio.so
0x00002b59acd0e000	47K	/home/dev/jdk1.6.0_20/jre/lib/amd64/jli/libjli.so
0x00002b59ace1a000	11222K	/home/dev/jdk1.6.0_20/jre/lib/amd64/server/libjvm.so

[dev@pingtai config]$ jinfo 434
Attaching to process ID 434, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 16.3-b01
Java System Properties:

java.runtime.name = Java(TM) SE Runtime Environment
elasticsearch =
sun.boot.library.path = /home/dev/jdk1.6.0_20/jre/lib/amd64
java.vm.version = 16.3-b01
es.path.home = /srv/elasticsearch/es_svn/es_deploy_0.16.2
java.vm.vendor = Sun Microsystems Inc.
java.vendor.url = http://java.sun.com/
path.separator = :
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
file.encoding.pkg = sun.io
sun.java.launcher = SUN_STANDARD
user.country = US
... ...

本文来自: jps&jstack&jmap

, , ,

热门话题，时间及空目录的处理

<Date: 2011-07-01> <Author: medcl> <Category: Hadoop, Linux> 发表评论

 

先查看hadoop目录的文件数，然后再决定是不是在input里面加上该目录
[dev@platformB dailyrawdata]$  fs -ls / |wc -l
3

计算时间的方法
[dev@platformB dailyrawdata]$ lastdate=20110619
[dev@platformB dailyrawdata]$ echo $lastdate
20110619
[dev@platformB dailyrawdata]$ echo `date --date "-d $lastdate + 1day" +"%Y%m%d" `
20110620

[dev@platformB dailyrawdata]$ echo D9=`date --date "now -20 day" +"%Y%m%d"`
D9=20110530

 

[dev@platformB dailyrawdata]$ D1=`date --date "now" +"%Y/%m/%d"`
[dev@platformB dailyrawdata]$ echo $D1
2011/06/20

注：等号后面不能有空格，如下面：

[dev@platformB dailyrawdata]$ D1= `date --date "now" +"%Y/%m/%d"`
-bash: 2011/06/20: No such file or directory

 

拷贝今天的文件到指定目录

DAYSTR=`date --date "now" +"%Y/%m/%d"`

hadoop fs -copyFromLocal dailyrawdata/* /trendingtopics/data/raw/$DAYSTR

 

慢着，当目录下文件为空的时候，Hadoop Stream Job的根据你指定的Input Pattern找不到文件的时候会抛异常，结果就造成了Job的失败。

找了半天也没有找到好的办法(那个知道比较好的办法，还请不吝赐教啊)，只能先判断目录是否为空，为空则将文件夹重定向到一个空文件。

#touch blank file
BLANK="/your folder/temp/blank"
hadoop fs -touchz $BLANK

#define a function to check hdfs files
function check_hdfs_files(){

#run hdfs command to check the files
hadoop fs -ls $1 &>/dev/null

#if file match is zero
#check file exists
if  [ $? -ne 0 ]
then
eval "$2=$BLANK"
echo "can't find any files,use blank file instead"
fi

return $?
}

 

--

D0=`date --date "now" +"/your folder/%Y/%m/%d/${APPNAME}-${TENANT}*"`
D1=`date --date "now -1 day" +"/your folder/%Y/%m/%d/$APPNAME-$TENANT*"`

#check file exists
check_hdfs_files $D0 "D0"
check_hdfs_files $D1 "D1"

本文来自: 热门话题，时间及空目录的处理

, , ,

shell脚本：/bin/sh^M: bad interpreter: No such file or directory。

<Date: 2011-06-15> <Author: medcl> <Category: Linux> 发表评论

 

今天再一次碰到了这个问题。

在Linux中执行.sh脚本，异常/bin/^M: bad interpreter: No such file or directory。
分析：这是不同系统编码格式引起的：在windows系统中编辑的.sh文件可能有不可见字符，所以在Linux系统下执行会报以上异常信息。
解决：1）在windows下转换：
        利用一些编辑器如UltraEdit或EditPlus等工具先将脚本编码转换，再放到Linux中执行。转换方式如下（UltraEdit）：File-->Conversions-->DOS->UNIX即可。
       2）也可在Linux中转换：
         首先要确保文件有可执行权限
         #sh>chmod a+x filename
          然后修改文件格式
          #sh>vi filename
          利用如下命令查看文件格式
         :set ff  或 :set fileformat
          可以看到如下信息
          fileformat=dos 或 fileformat=unix
           利用如下命令修改文件格式
          :set ff=unix 或 :set fileformat=unix
          :wq (存盘退出)
          最后再执行文件
          #sh>./filename

本文来自: shell脚本：/bin/sh^M: bad interpreter: No such file or directory。

nginx升级1.0及自定义server-header

<Date: 2011-04-12> <Author: medcl> <Category: Linux> 发表评论

今天发布了nginx1.0，把我的vps的也升级下，顺便修改下header信息，移除了版本号。

移除header信息里面nginx的版本号

vi nginx.conf
#http{}里面加：
 
server_tokens off;
 
vi /usr/local/nginx/conf/fastcgi_params
vi /usr/local/nginx/conf/fcgi.conf
修改下面的信息：
fastcgi_param  SERVER_SOFTWARE
 
--
nginx-admin reload

1.0升级步骤

cd /tmp
wget http://nginx.org/download/nginx-1.0.0.tar.gz
tar vxzf nginx-1.0.0.tar.gz
cd nginx-1.0.0 
 
 ./configure --user=www --group=www --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module
make && make install
 
#安装程序自动升级，原来的自动重命名为nginx.old
 
/usr/local/nginx/sbin/nginx -v
nginx: nginx version: nginx/1.0.0

修改header里面server的名称，让你的servername更加个性：

 vi src/http/ngx_http__filter_module.c
 48 static char ngx_http_server_string[] = "Server: INFINITBYTE" CRLF;
 49 static char ngx_http_server_full_string[] = "Server: INFINITBYTE" CRLF;

重新编译安装

效果如下,是不是很酷？：

>curl --head medcl.net
HTTP/1.1 200 OK
Server: INFINITBYTE
Date: Tue, 12 Apr 2011 10:47:06 GMT
Content-Type: text/html
Content-Length: 0
Last-Modified: Fri, 23 Apr 2010 05:43:36 GMT
Connection: keep-alive
Accept-Ranges: bytes

本文来自: nginx升级1.0及自定义server-header

, ,

使用dropbear代替sshd

<Date: 2011-04-11> <Author: medcl> <Category: Linux> 发表评论 
touch /etc/ssh/sshd_not_to_be_run
 
wget http://matt.ucc.asn.au/dropbear/dropbear-0.52.tar.gz
tar -xvzf dropbear-0.52.tar.gz
cd dropbear-0.52
./configure
make && make install
 
 
mkdir /etc/dropbear
/usr/local/bin/dropbearkey -t dss -f /etc/dropbear/dropbear_dss_host_key
/usr/local/bin/dropbearkey -t rsa -s 4096 -f /etc/dropbear/dropbear_rsa_host_key
 
vi /etc/rc.local
Add:
/usr/local/sbin/dropbear
 
#指定监听端口
/usr/local/sbin/dropbear –p 2222

本文来自: 使用dropbear代替sshd

,

Linux kernel 的 sendfile 是如何提高性能的

<Date: 2011-04-11> <Author: medcl> <Category: Linux> 发表评论

转：http://www.vpsee.com/2009/07/linux-sendfile-improve-performance/

现在流行的 web 服务器里面都提供 选项用来提高服务器性能，那到底 是什么，怎么影响性能的呢？ 实际上是 2.0+ 以后的推出的一个系统调用，web 服务器可以通过调整自身的配置来决定是否利用 sendfile 这个系统调用。先来看一下不用 sendfile 的传统网络传输过程：

read(file, tmp_buf, len);
write(socket, tmp_buf, len);

硬盘 >> buffer >> user buffer >> socket buffer >> 协议栈

一般来说一个网络应用是通过读硬盘数据，然后写数据到 socket 来完成网络传输的。上面2行用代码解释了这一点，不过上面2行简单的代码掩盖了底层的很多操作。来看看底层是怎么执行上面2行代码的：

1、系统调用 read() 产生一个上下文切换：从 user mode 切换到 kernel mode，然后 DMA 执行拷贝，把文件数据从硬盘读到一个 kernel buffer 里。
2、数据从 kernel buffer 拷贝到 user buffer，然后系统调用 read() 返回，这时又产生一个上下文切换：从kernel mode 切换到 user mode。
3、系统调用 write() 产生一个上下文切换：从 user mode 切换到 kernel mode，然后把步骤2读到 user buffer 的数据拷贝到 kernel buffer（数据第2次拷贝到 kernel buffer），不过这次是个不同的 kernel buffer，这个 buffer 和 socket 相关联。
4、系统调用 write() 返回，产生一个上下文切换：从 kernel mode 切换到 user mode（第4次切换了），然后 DMA 从 kernel buffer 拷贝数据到协议栈（第4次拷贝了）。

上面4个步骤有4次上下文切换，有4次拷贝，我们发现如果能减少切换次数和拷贝次数将会有效提升性能。在kernel 2.0+ 版本中，系统调用 sendfile() 就是用来简化上面步骤提升性能的。sendfile() 不但能减少切换次数而且还能减少拷贝次数。

再来看一下用 sendfile() 来进行网络传输的过程：

sendfile(socket, file, len);

硬盘 >> kernel buffer (快速拷贝到kernel socket buffer) >> 协议栈

1、系统调用 sendfile() 通过 DMA 把硬盘数据拷贝到 kernel buffer，然后数据被 kernel 直接拷贝到另外一个与 socket 相关的 kernel buffer。这里没有 user mode 和 kernel mode 之间的切换，在 kernel 中直接完成了从一个 buffer 到另一个 buffer 的拷贝。
2、DMA 把数据从 kernel buffer 直接拷贝给协议栈，没有切换，也不需要数据从 user mode 拷贝到 kernel mode，因为数据就在 kernel 里。

步骤减少了，切换减少了，拷贝减少了，自然性能就提升了。这就是为什么说在 配置文件里打开 sendfile on 选项能提高 web server性能的原因。

本文来自: Linux kernel 的 sendfile 是如何提高性能的

, ,

mysql-bin.000001-x

<Date: 2011-04-10> <Author: medcl> <Category: Linux, mysql> 发表评论

发达了，-bin占了我9g空间啊，大汗一把：
20K .000001
700K mysql-bin.000002
4.0K mysql-bin.000003
4.0K mysql-bin.000004
4.0K mysql-bin.000005
4.0K mysql-bin.000006
4.0K mysql-bin.000007
4.0K mysql-bin.000008
4.0K mysql-bin.000009
4.0K mysql-bin.000010
11M mysql-bin.000011
63M mysql-bin.000012
368K mysql-bin.000013
256K mysql-bin.000014
9.5M mysql-bin.000015
1.8M mysql-bin.000016
11M mysql-bin.000017
508K mysql-bin.000018
45M mysql-bin.000019
146M mysql-bin.000020
86M mysql-bin.000021
2.2M mysql-bin.000022
2.6M mysql-bin.000023
124K mysql-bin.000024
8.7M mysql-bin.000025
4.9M mysql-bin.000026
428K mysql-bin.000027
372K mysql-bin.000028
123M mysql-bin.000029
17M mysql-bin.000030
4.0K mysql-bin.000031
700K mysql-bin.000032
454M mysql-bin.000033
14M mysql-bin.000034
841M mysql-bin.000035
388M mysql-bin.000036
26M mysql-bin.000037
728K mysql-bin.000038
216M mysql-bin.000039
228M mysql-bin.000040
294M mysql-bin.000041
14M mysql-bin.000042
155M mysql-bin.000043
13M mysql-bin.000044
884K mysql-bin.000045
13M mysql-bin.000046
434M mysql-bin.000047
206M mysql-bin.000048
171M mysql-bin.000049
528K mysql-bin.000050
61M mysql-bin.000051
330M mysql-bin.000052
199M mysql-bin.000053
64M mysql-bin.000054
436K mysql-bin.000055
17M mysql-bin.000056
340K mysql-bin.000057
116M mysql-bin.000058
18M mysql-bin.000059
23M mysql-bin.000060
188M mysql-bin.000061
40K mysql-bin.000062
277M mysql-bin.000063
4.0K mysql-bin.000064
35M mysql-bin.000065
258M mysql-bin.000066
147M mysql-bin.000067
79M mysql-bin.000068
4.0K mysql-bin.000069
4.0K mysql-bin.000070
434M mysql-bin.000071
4.0K mysql-bin.000072
99M mysql-bin.000073
52M mysql-bin.000074
620K mysql-bin.000075
428M mysql-bin.000076
788K mysql-bin.000077
4.0K mysql-bin.000078
4.0K mysql-bin.000079
126M mysql-bin.000080
137M mysql-bin.000081
428K mysql-bin.000082
31M mysql-bin.000083
4.0K mysql-bin.index
阅读这篇文章的其余部分 »

本文来自: mysql-bin.000001-x

,

php-cgi内存优化

<Date: 2011-04-09> <Author: medcl> <Category: Linux> 发表评论

php-cgi内存优化

优化前：

26577 root 15 0 93080 5020 988 S 0.0 1.0 0:00.64 php-cgi Kill Process 
26578 www 16 0 130m 48m 4756 S 0.0 9.5 0:00.63 php-cgi Kill Process 
26581 www 16 0 131m 50m 4992 S 0.0 9.9 0:00.87 php-cgi Kill Process 
26583 www 16 0 134m 50m 5236 S 0.0 9.9 0:00.82 php-cgi Kill Process 
26586 www 16 0 133m 51m 6352 S 0.0 10.0 0:00.72 php-cgi Kill Process 
26587 www 18 0 120m 39m 4724 S 0.0 7.6 0:00.80 php-cgi Kill Process 
26588 www 16 0 134m 50m 5272 S 0.0 9.9 0:00.87 php-cgi Kill Process 
26589 www 15 0 130m 48m 4756 S 0.0 9.5 0:00.64 php-cgi Kill Process 
26590 www 16 0 125m 43m 5224 S 0.0 8.4 0:00.65 php-cgi Kill Process

修改/usr/local/php/etc下的php-fpm.conf和php.ini文件，去掉不需要的扩展

优化后：

19488 root 15 0 82508 3736 980 S 0.0 0.7 0:00.04 php-cgi Kill Process 
19489 www 18 0 82548 4772 1888 S 0.0 0.9 0:00.00 php-cgi Kill Process 
19490 www 18 0 82508 4608 1804 S 0.0 0.9 0:00.00 php-cgi Kill Process 
19491 www 22 0 82508 3700 940 S 0.0 0.7 0:00.00 php-cgi Kill Process 
19492 www 18 0 82548 4772 1888 S 0.0 0.9 0:00.00 php-cgi Kill Process 
19493 www 18 0 82548 4776 1888 S 0.0 0.9 0:00.00 php-cgi Kill Process 
19494 www 18 0 82548 4776 1888 S 0.0 0.9 0:00.00 php-cgi Kill Process 
19497 www 18 0 82548 4772 1888 S 0.0 0.9 0:00.00 php-cgi Kill Process 
19498 www 21 0 82508 3700 940 S 0.0 0.7 0:00.00 php-cgi Kill Process 
19499 www 21 0 82508 3700 940 S 0.0 0.7 0:00.00 php-cgi Kill Process 
19500 www 25 0 82508 3700 940 S 0.0 0.7 0:00.00 php-cgi Kill Process

只要一个php-cgi进程占用的内存超过 %1 就把它kill掉

#!/bin/sh
 
PIDS=`ps aux|grep php-cgi|grep -v grep|awk '{if($4>=1)print $2}'`
 
for PID in $PIDS
do
echo `date +%F....%T` >> /tmp/phpkill.log
echo $PID >> /tmp/phpkill.log
kill -9  $PID
done

放crontab里面，每5分钟执行一次

mv killphpcgi.sh /usr/local/php/
crontab -e
 
*/5 * * * * /usr/local/php/killphpcgi.sh
 
crontab -l

htop挺不错,可以很方便对cpu占用率、内存占用率等进行排序，推荐一下

yum install htop
wget http://downloads.sourceforge.net/project/htop/htop/0.9/htop-0.9.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fhtop%2F&ts=1302575211&use_mirror=cdnetworks-kr-1

本文来自: php-cgi内存优化

,