制作Zeppelin的Parcel包和csd文件

配置java，maven等环境变量

java:

 export JAVA_HOME=/usr/local/java
 export PATH=$JAVA_HOME/bin:$PATH
 export CLASSPATH=$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tool.jar:$CLASSPATH

复制

maven:

 export MVN_HOME=/usr/local/maven
 export PATH=$MVN_HOME/bin:$PATH

复制

制作 Zeppelin 的 Parcel 包并校验

注意：

此脚本默使用Livy-0.5.0 ，Zeppelin-0.8.0版本，要求Python版本2.7/3.3或更高版本。如需修改Zeppelin版本，修改build.sh脚本中下面变量即可：

 ...
 # Zeppelin安装包下载路径
 ZEPPELIN_URL=http://apache.mirror.anlx.net/zeppelin/zeppelin-0.8.0/zeppelin-0.8.0-bin-all.tgz
 # Zeppelin安装包的MD5值下载路径
 ZEPPELIN_MD5="ee6875ac47492e37899dea103b34d176"
 # 或直接使用下面配置跳过MD5校验
 #ZEPPELIN_MD5="$(md5sum $( basename $ZEPPELIN_URL ))"
 # Zeppelin 版本
 ZEPPELIN_VERSION=0.8.1
 ...
 # Linux系统，将el7修改为对应的系统版本即可
 zeppelin_parcel_name="$zeppelin_parcel_folder-el7.parcel"
 ...

复制

运行制作parcel 包脚本

 [root@cdh01 ~]# mkdir -p ~/github/cloudera/
 [root@cdh01 ~]# cd ~/github/cloudera/ 
 [root@cdh01 cloudera]# git clone https://github.com/alexjbush/livy_zeppelin_cdh_csd_parcels.git
 [root@cdh01 cloudera]# cd livy_zeppelin_cdh_csd_parcels    
 [root@cdh01 livy_zeppelin_cdh_csd_parcels]# sh build.sh parcel

复制

日志

 ...
 + java -jar cm_ext/validator/target/validator.jar -f ./ZEPPELIN-0.8.1_build/ZEPPELIN-0.8.1-el7.parcel
 Validating: ./ZEPPELIN-0.8.1_build/ZEPPELIN-0.8.1-el7.parcel
 Validating: ZEPPELIN-0.8.1/meta/parcel.json
 Validation succeeded.
 + python cm_ext/make_manifest/make_manifest.py ./ZEPPELIN-0.8.1_build
 Scanning directory: ./ZEPPELIN-0.8.1_build
 Found parcel ZEPPELIN-0.8.1-el7.parcel

复制

遇到的问题：

问题一：

 [ERROR] Plugin org.apache.maven.plugins:maven-jar-plugin:2.3.1 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-jar-plugin:jar:2.3.1: Could not transfer artifact org.apache.maven.plugins:maven-jar-plugin:pom:2.3.1 from/to central (https://repo.maven.apache.org/maven2): Connect to repo.maven.apache.org:443 [repo.maven.apache.org/151.101.40.215] failed: Connection refused -> [Help 1]

复制

解决办法：

修改maven安装目录下的conf/setting.xml文件，将中央仓库地址阿里云镜像地址。

 <mirror>
   <id>alimaven</id>
   <name>aliyun maven</name>
   <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
   <mirrorOf>central</mirrorOf>       
 </mirror>

复制

问题二：

 [INFO] Cloudera Service Schemas ........................... FAILURE [  8.483 s]
 [INFO] Schema Validator ................................... SKIPPED
 [INFO] Cloudera Manager Extensibility Tools ............... SKIPPED
 [INFO] ------------------------------------------------------------------------
 [INFO] BUILD FAILURE
 [INFO] ------------------------------------------------------------------------
 [INFO] Total time: 8.820 s
 [INFO] Finished at: 2019-05-26T15:08:45+08:00
 [INFO] Final Memory: 11M/57M
 [INFO] ------------------------------------------------------------------------
 [ERROR] Failed to execute goal on project cloudera-manager-schema: Could not resolve dependencies for project com.cloudera.cmf.schema:cloudera-manager-schema:jar:5.12.0: Failed to collect dependencies at commons-cli:commons-cli:jar:1.3-cloudera-pre-r1439998: Failed to read artifact descriptor for commons-cli:commons-cli:jar:1.3-cloudera-pre-r1439998: Could not transfer artifact commons-cli:commons-cli:pom:1.3-cloudera-pre-r1439998 from/to cloudera-external (https://repository.cloudera.com/artifactory/ext-release-local/): Remote host closed connection during handshake: SSL peer shut down incorrectly -> [Help 1]

复制

解决办法：

进入maven仓库下，手动动下载对应的commons-cli包。

 [root@cdh01 ~]# cd ~/.m2/repository/
 [root@cdh01 repository]# cd commons-cli/commons-cli/1.3-cloudera-pre-r1439998
 [root@cdh01 1.3-cloudera-pre-r1439998]# wget https://repository.cloudera.com/artifactory/ext-release-local/commons-cli/commons-cli/1.3-cloudera-pre-r1439998/commons-cli-1.3-cloudera-pre-r1439998.pom
 [root@cdh01 1.3-cloudera-pre-r1439998]# wget https://repository.cloudera.com/artifactory/ext-release-local/commons-cli/commons-cli/1.3-cloudera-pre-r1439998/commons-cli-1.3-cloudera-pre-r1439998.jar

复制

制作 Zeppelin 的CSD文件

 [root@cdh01 livy_zeppelin_cdh_csd_parcels]# sh build.sh csd
 + set -e
 + CM_EXT_BRANCH=cm5-5.12.0
 + LIVY_URL=http://apache.mirror.anlx.net/incubator/livy/0.6.0-incubating/apache-livy-0.6.0-incubating-bin.zip
 + LIVY_MD5=55f966dc4b25ba595428dbc4634fac01
 + LIVY_VERSION=0.6.0
 + ZEPPELIN_URL=http://apache.mirror.anlx.net/zeppelin/zeppelin-0.8.1/zeppelin-0.8.1-bin-all.tgz
 + ZEPPELIN_MD5=ee6875ac47492e37899dea103b34d176
 + ZEPPELIN_VERSION=0.8.1
 + livy_service_name=LIVY
 ++ tr '[:upper:]' '[:lower:]'
 ++ echo LIVY
 + livy_service_name_lower=livy
 ++ basename http://apache.mirror.anlx.net/incubator/livy/0.6.0-incubating/apache-livy-0.6.0-incubating-bin.zip
 + livy_archive=apache-livy-0.6.0-incubating-bin.zip
 ++ basename apache-livy-0.6.0-incubating-bin.zip .zip
 + livy_folder=apache-livy-0.6.0-incubating-bin
 + livy_parcel_folder=LIVY-0.6.0
 + livy_parcel_name=LIVY-0.6.0-el7.parcel
 + livy_built_folder=LIVY-0.6.0_build
 + livy_csd_build_folder=livy_csd_build
 + zeppelin_service_name=ZEPPELIN
 ++ tr '[:upper:]' '[:lower:]'
 ++ echo ZEPPELIN
 + zeppelin_service_name_lower=zeppelin
 ++ basename http://apache.mirror.anlx.net/zeppelin/zeppelin-0.8.1/zeppelin-0.8.1-bin-all.tgz
 + zeppelin_archive=zeppelin-0.8.1-bin-all.tgz
 ++ basename zeppelin-0.8.1-bin-all.tgz .tgz
 + zeppelin_folder=zeppelin-0.8.1-bin-all
 + zeppelin_parcel_folder=ZEPPELIN-0.8.1
 + zeppelin_parcel_name=ZEPPELIN-0.8.1-el7.parcel
 + zeppelin_built_folder=ZEPPELIN-0.8.1_build
 + zeppelin_csd_build_folder=zeppelin_csd_build
 + case $1 in
 + build_livy_csd
 + JARNAME=LIVY-0.6.0.jar
 + '[' -f LIVY-0.6.0.jar ']'
 + rm -rf livy_csd_build
 + cp -rf ./livy-csd-src livy_csd_build
 + sed -i -e s/%SERVICENAME%/LIVY/ livy_csd_build/descriptor/service.sdl
 + sed -i -e s/%SERVICENAMELOWER%/livy/ livy_csd_build/descriptor/service.sdl
 + sed -i -e s/%SERVICENAMELOWER%/livy/ livy_csd_build/scripts/control.sh
 + java -jar cm_ext/validator/target/validator.jar -s livy_csd_build/descriptor/service.sdl -l 'SPARK_ON_YARN SPARK2_ON_YARN'
 Validating: livy_csd_build/descriptor/service.sdl
 Validation succeeded.
 + jar -cvf ./LIVY-0.6.0.jar -C livy_csd_build .
 added manifest
 adding: images/(in = 0) (out= 0)(stored 0%)
 adding: images/icon.png(in = 1166) (out= 700)(deflated 39%)
 adding: descriptor/(in = 0) (out= 0)(stored 0%)
 adding: descriptor/service.sdl(in = 9551) (out= 2390)(deflated 74%)
 adding: aux/(in = 0) (out= 0)(stored 0%)
 adding: aux/client/(in = 0) (out= 0)(stored 0%)
 adding: aux/client/livy-env.sh(in = 21) (out= 23)(deflated -9%)
 adding: scripts/(in = 0) (out= 0)(stored 0%)
 adding: scripts/control.sh(in = 3598) (out= 1169)(deflated 67%)
 + build_zeppelin_csd
 + JARNAME=ZEPPELIN-0.8.1.jar
 + '[' -f ZEPPELIN-0.8.1.jar ']'
 + rm -rf zeppelin_csd_build
 + cp -rf ./zeppelin-csd-src zeppelin_csd_build
 + sed -i -e s/%SERVICENAME%/ZEPPELIN/ zeppelin_csd_build/descriptor/service.sdl
 + sed -i -e s/%SERVICENAMELOWER%/zeppelin/ zeppelin_csd_build/descriptor/service.sdl
 + sed -i -e s/%LIVYSERVICENAME%/LIVY/ zeppelin_csd_build/descriptor/service.sdl
 + sed -i -e s/%LIVYSERVICENAMELOWER%/livy/ zeppelin_csd_build/descriptor/service.sdl
 + sed -i -e s/%SERVICENAMELOWER%/zeppelin/ zeppelin_csd_build/scripts/control.py
 + sed -i -e s/%LIVYSERVICENAMELOWER%/livy/ zeppelin_csd_build/scripts/control.py
 + java -jar cm_ext/validator/target/validator.jar -s zeppelin_csd_build/descriptor/service.sdl -l LIVY
 Validating: zeppelin_csd_build/descriptor/service.sdl
 Validation succeeded.
 + jar -cvf ./ZEPPELIN-0.8.1.jar -C zeppelin_csd_build .
 added manifest
 adding: images/(in = 0) (out= 0)(stored 0%)
 adding: images/icon.png(in = 1166) (out= 573)(deflated 50%)
 adding: descriptor/(in = 0) (out= 0)(stored 0%)
 adding: descriptor/service.sdl(in = 14376) (out= 3057)(deflated 78%)
 adding: aux/(in = 0) (out= 0)(stored 0%)
 adding: aux/client/(in = 0) (out= 0)(stored 0%)
 adding: aux/client/interpreter.livy.json(in = 2362) (out= 600)(deflated 74%)
 adding: aux/client/zeppelin-env.sh(in = 21) (out= 23)(deflated -9%)
 adding: aux/client/interpreter.json(in = 1753) (out= 518)(deflated 70%)
 adding: aux/client/shiro.ini(in = 0) (out= 0)(stored 0%)
 adding: scripts/(in = 0) (out= 0)(stored 0%)
 adding: scripts/test_control.py(in = 565) (out= 273)(deflated 51%)
 adding: scripts/control.py(in = 7640) (out= 2200)(deflated 71%)

复制

查看Parcel包和csd文件

查看Parcel包

 [root@hadoop01 livy_zeppelin_cdh_csd_parcels]# ll ZEPPELIN-0.8.1_build LIVY-0.6.0_build
 LIVY-0.6.0_build:
 total 93748
 -rw-r--r--. 1 root root 95993481 May 26 16:30 LIVY-0.6.0-el7.parcel
 -rw-r--r--. 1 root root      311 May 26 16:30 manifest.json

 ZEPPELIN-0.8.1_build:
 total 968732
 -rw-r--r--. 1 root root       316 May 26 16:33 manifest.json
 -rw-r--r--. 1 root root 991970787 May 26 16:32 ZEPPELIN-0.8.1-el7.parcel

复制

查看csd文件

 total 1063740
 -rw-r--r--. 1 root root   96137660 May 24 20:36 apache-livy-0.6.0-incubating-bin.zip
 -rw-r--r--. 1 root root       6362 May 26 14:40 build.sh
 drwxr-xr-x. 6 root root       4096 May 26 13:42 cm_ext
 -rw-r--r--. 1 root root      63264 May 26 13:23 kerberos.png
 drwxr-xr-x. 8 root root       4096 May 26 16:03 LIVY-0.6.0
 drwxr-xr-x. 2 root root       4096 May 26 16:30 LIVY-0.6.0_build
 -rw-r--r--. 1 root root       5607 May 26 16:36 LIVY-0.6.0.jar
 drwxr-xr-x. 6 root root       4096 May 26 16:35 livy_csd_build
 drwxr-xr-x. 6 root root       4096 May 26 13:23 livy-csd-src
 drwxr-xr-x. 3 root root       4096 May 26 13:23 livy-parcel-src
 -rw-r--r--. 1 root root       4391 May 26 13:23 README.md
 drwxr-xr-x. 9  503 wheel      4096 May 26 16:31 ZEPPELIN-0.8.1
 -rw-r--r--. 1 root root  992975720 May 25 12:25 zeppelin-0.8.1-bin-all.tgz
 drwxr-xr-x. 2 root root       4096 May 26 16:33 ZEPPELIN-0.8.1_build
 -rw-r--r--. 1 root root       9133 May 26 16:36 ZEPPELIN-0.8.1.jar
 drwxr-xr-x. 6 root root       4096 May 26 16:36 zeppelin_csd_build
 drwxr-xr-x. 6 root root       4096 May 26 13:23 zeppelin-csd-src
 drwxr-xr-x. 3 root root       4096 May 26 13:23 zeppelin-parcel-src

复制