当前位置:   article > 正文

Ambari--从零开始集成自定义组件_ambari添加自定义组件

ambari添加自定义组件

 

  1. rpm包制作

作用:集成所有docker依赖离线安装包,安装脚本。

 

1.1 安装工具

安装rpm-build

yum install rpm-build -y

 

安装rpmdevtools

yum install rpmdevtools -y

 

1.2 创建工作空间

创建工作空间

rpmdev-setuptree -y

查看工作空间

 

创建文件夹

mkdir -p /root/rpmbuild/SOURCES/zoo/{bin,conf,lib}

1.3 拷贝所需文件

将docker安装包和安装脚本拷贝至lib文件夹下

Install.sh为安装脚本,内容如下

#!/bin/bash

function install_docker(){

 

#卸载cgroups

yum remove libcgroup -y

 

#关闭防火墙

systemctl stop firewalld

systemctl disable firewalld

 

#关闭selinux

setenforce 0

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

 

#开启服务器转发

modprobe br_netfilter

echo "net.ipv4.ip_forward = 1

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.conf

sysctl -p

 

#关闭swap

swapoff -a

sed -i '/swap/s/^/#/g' /etc/fstab

 

#开启cgroups

sed -i '/GRUB_CMDLINE_LINUX/d' /etc/default/grub

echo 'GRUB_CMDLINE_LINUX_DEFAULT="cgroup_enable=memory swapaccount=1"

GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"' >> /etc/default/grub

 

#安装docker

yum localinstall /usr/hdp/3.0.1.0-187/zoo/lib/docker/*.rpm -y

systemctl restart docker

 

}

install_docker

修改文件权限

chmod +x install.sh

进入/root/rpmbuild/SOURCES目录

打包zoo压缩包

tar -cvzf zoo.tar.gz zoo/

 

1.4 编辑spec文件打包

进入/root/rpmbuild/SPECS目录

编写zoo.spec文件,内容如下

其中%install为执行安装的步骤

 

Name:         zoo

Version:        1.0

# Avoid some silly dependencies and allow

# symlink in the init.d folder

AutoReqProv:    no

Release:        0

Summary:        A unified Data Analytics and AI platform for distributed TensorFlow, Keras and PyTorch on Apache Spark/Flink & Ray.

License:        Apache 2.0

Group:          Applications/Internet

Source0:        zoo.tar.gz

Prefix:         /usr/hdp/3.0.1.0-187

BuildArch:      noarch

BuildRoot:      %{_tmppath}/%{name}-root

Provides:       zoo

Requires(post): /sbin/chkconfig, /sbin/service

%description

A unified Data Analytics and AI platform for distributed TensorFlow, Keras and PyTorch on Apache Spark/Flink & Ray

 

 

%prep

#打包时不丢弃debug信息

%define __strip /bin/true

#打包之前不要对目标文件做strip

%define __os_install_post %{nil}

#禁止自动搜索依赖关系

#Autoreq:no

#Autoprov:no

 

%build

 

%install

rm -rf $RPM_BUILD_ROOT

mkdir -p $RPM_BUILD_ROOT/usr/hdp/3.0.1.0-187/zoo

mkdir -p $RPM_BUILD_ROOT/etc/init.d

mkdir -p %{_localstatedir}/lib/zoo

tar xfz $RPM_SOURCE_DIR/zoo.tar.gz

cp -r $RPM_BUILD_DIR/zoo/* $RPM_BUILD_ROOT/usr/hdp/3.0.1.0-187/zoo

#%pre

 

%pre -p /bin/sh

 

%preun

 

%clean

 

#%post

%post -p /bin/sh

 

%files

%defattr(-,root,root,-)

/usr/hdp/3.0.1.0-187/zoo

 

%changelog

* Tue Aug 19 2014 - daniel.beneyto at abiquo dot com

- Initial release.

 

 

1.5 执行打包命令

rpmbuild -ba zoo.spec

查看rpm包

cd /root/rpmbuild/RPMS/noarch

 

 

 

  1. rpm本地缓存及仓库构建

作用:  使ambari执行yum install时使用本地rpm包

2.1 安装工具

安装createrepo

yum install createrepo -y

2.2 创建本地yum缓存地址

mkdir -p /var/www/html/zoo-repo/

复制rpm包至/var/www/html/zoo-repo/文件夹下

创建yum缓存文件

cd /var/www/html/zoo-repo/

createrepo ./

 

2.3 配置repo文件

在/etc/yum.repos.d/文件夹下创建zoo.repo,内容如下

 

[zoo]

name=zoo

baseurl=file:///var//www/html/zoo-repo

gpagcheck=0

enabled=1

 

2.4 清除当前yum缓存并重新构建

yum clean all

yum makecache

 

 

  1. 镜像制作

节点:10.180.210.37

作用:Intel Analytics-Zoo环境及Jupyter环境

3.1 Dockerfile

改动  删除启动脚本start-notebook.sh

修改entrypoint和cmd  支持参数传入

Dockerfile如下,没有配置环境变量,完整Dockerfile见10.180.210.37:/home/zoo/dockerfile/Dockerfile

FROM centos:centos7.4.1708

WORKDIR /

RUN  mkdir -p /opt/workspace/

ADD  anaconda3.tar.gz /root/

ADD  test.tar.gz /opt/workspace/

COPY hadoop /usr/hdp/3.0.1.0-187/hadoop

COPY spark2 /usr/hdp/3.0.1.0-187/spark2

COPY java   /usr/jdk64/java

COPY etc/hadoop /etc/hadoop

COPY hadoop-hdfs /usr/hdp/3.0.1.0-187/hadoop-hdfs

COPY hadoop-mapreduce /usr/hdp/3.0.1.0-187/hadoop-mapreduce

COPY hadoop-yarn /usr/hdp/3.0.1.0-187/hadoop-yarn

COPY usr /usr/hdp/3.0.1.0-187/usr

ENTRYPOINT ["jupyter","notebook","--allow-root","--no-browser","--ip='*'","--notebook-dir='/opt/workspace'"]

CMD ["--port=12345","--NotebookApp.token=qwer1234"]

3.2 镜像构建和保存

docker build  -t zoo-insight-v4:v1.0 .

docker save zoo-insight-v4:v1.0 > zoo-InsightV4.tar

 

  1. Ambari 集成

4.1 创建组件配置文件夹

在insightV4 manager节点执行如下命令

cd /var/lib/ambari-server/resources/stacks/HDP/3.0/services

创建文件夹

mkdir ZOO

创建如下文件夹

cd ZOO

下列所有操作均在ZOO文件夹及子文件夹内进行

mkdir -p  configuration quicklinks package/scripts

4.2 主配置文件metainfo.xml

创建组件主配置文件metainfo.xml

内容如下

<?xml version="1.0"?>

 

<!-- AnalyticsZoo metainfo -->

 

<metainfo>

    <schemaVersion>1.9.0</schemaVersion>

    <services>

        <service>

 

            <!--  组件名称-->

            <name>ZOO</name>

            <!--  组件显示名称-->

            <displayName>AnalyticsZoo</displayName>

            <!--  组件显示简介-->

            <comment>Analytics-Zoo是基于spark的大数据分析+AI的端到端的分布式深度学习框架</comment>

            <!--  组件显示版本-->

    <version>1.0.0</version>

            <components>

                <component>

                    <name>ZOO</name>

                    <displayName>Analytics-Zoo</displayName>

<!-- 组件节点-->

                    <category>MASTER</category>

                    <!-- 组件节点个数-->

                    <cardinality>1</cardinality>

                    <versionAdvertised>true</versionAdvertised>

                    <commandScript>

                        <!-- 组件操作脚本 -->

                        <script>scripts/zoo.py</script>

                        <!-- 组件操作脚本语言 -->

                        <scriptType>PYTHON</scriptType>

                        <!-- 组件操作脚本执行超时时间 -->                        <timeout>1200</timeout>

                    </commandScript>

                    <logs>

                        <log>

                            <logId>analytics-zoo</logId>

                            <primary>true</primary>

                        </log>

                    </logs>

                </component>

            </components>

            <osSpecifics>

                <osSpecific>

                    <osFamily>any</osFamily>

                    <packages>

                        <package>

                            <name>zoo</name>

                        </package>

                    </packages>

                </osSpecific>

            </osSpecifics>

            <configuration-dependencies>

                <!-- 组件参数配置文件-->

                <config-type>analytics-zoo-env</config-type>

            </configuration-dependencies>

            <quickLinksConfigurations>

                <quickLinksConfiguration>

                    <!-- 组件快速链接配置文件-->

                    <fileName>quicklinks.json</fileName>

                    <default>true</default>

                </quickLinksConfiguration>

            </quickLinksConfigurations>

        </service>

    </services>

</metainfo>

 

4.3 配置组件参数配置文件

在configuration文件夹中创建analytics_zoo_env.xml内容如下:

<?xml version="1.0"?>

 

<?xml-stylesheet type-"text/xsl" href="configuration. xsl"?>

<!-- analytics-zoo-env -->

<configuration supports_adding_forbidden="true">

    <property>

        <name>notebook_port</name>

        <display-name>notebook_port</display-name>

        <value>12345</value>

        <description>notebook_port</description>

        <on-ambari-upgrade add="true"/>

    </property>

    <property>

        <name>notebook_token</name>

        <display-name>notebook_token</display-name>

        <description>notebook 密码</description>

        <value-attributes>

            <type>password</type>

            <overridable>false</overridable>

        </value-attributes>

        <on-ambari-upgrade add="true"/>

    </property>

</configuration>

 

4.4 Zoo容器化核心脚本,安装启停状态监测功能实现

在package/scripts文件夹下创建以下四个文件

params.py  读取页面配置及内置参数的脚本

start.sh   控制Zoo容器及docker服务启动的脚本

stop.sh   控制Zoo容器及docker服务停止的脚本

zoo.py    执行安装启停状态监测的脚本

   params.py内容如下

# -*- coding: utf-8 -*-

#!/usr/bin/env python

"""

Zoo Params configurations

"""

 

# from resource_management.libraries.functions.version import format_hdp_stack_version, compare_versions

from resource_management import *

from resource_management.core.logger import Logger

import os, re, json, functools

import resource_management.libraries.functions

from resource_management.libraries.functions.default import default

from resource_management.libraries.functions.format import format

from resource_management.libraries.script.script import Script

from os.path import join

 

Logger.initialize_logger()

config = Script.get_config()

# Logger.info("config:{0}".format(config))

Logger.info('Starting config params ...')

 

root_user = 'root'

 

analytics_zoo_env = config['configurations']['analytics_zoo_env']

Logger.info("analytics_zoo_env:{0}".format(analytics_zoo_env))

 

notebook_port = config['configurations']['analytics_zoo_env']['notebook_port']

Logger.info("notebook_port:{0}".format(notebook_port))

 

notebook_token = config['configurations']['analytics_zoo_env']['notebook_token']

Logger.info("notebook_token:{0}".format(notebook_token))

 

zoo_log_dir = '/var/log/zoo/'

zoo_log_file = 'zoo.log'

zoo_containerid_file = "zoo.containerid"

docker_pid_file = "docker.pid"

 

config = Script.get_config()

Zoo.py内容如下

# -*- coding: utf-8 -*-

 

from resource_management import *

from resource_management.core.logger import Logger

from resource_management.libraries.functions.check_process_status import check_process_status

from resource_management.core.resources.system import Execute, File

from resource_management.libraries.functions.show_logs import show_logs

import os

import socket

 

 

class Zoo(Script):

    def configure(self, env):

        import params

        Logger.info('Configure Starting Config Params ...')

        env.set_params(params)

        Logger.info('Configure Config Params.')

 

    def install(self, env):

        import params

        env.set_params(params)

        Logger.info('Starting Install Analytics-Zoo ...')

        self.install_packages(env)

        install_zoo_cmd = "/usr/hdp/3.0.1.0-187/zoo/lib/install.sh".format()

        Execute(install_zoo_cmd,user=params.root_user)

        Logger.info('Installed Analytics-Zoo!')

 

    def start(self, env):

        import params

        env.set_params(params)

        Logger.info('Starting Analytics-Zoo ...')

        notebook_port = params.notebook_port

        notebook_token = params.notebook_token

        start_zoo_cmd = "/var/lib/ambari-server/resources/stacks/HDP/3.0/services/ZOO/package/scripts/start.sh " +notebook_port + " " + notebook_token

        Execute(start_zoo_cmd,user=params.root_user)

        Logger.info('Started Analytics-Zoo !')

 

    def stop(self, env):

        stop_zoo_cmd = "/var/lib/ambari-server/resources/stacks/HDP/3.0/services/ZOO/package/scripts/stop.sh"

        os.system(stop_zoo_cmd)

        Logger.info('Stopping Analytics-Zoo')

 

 

    def status(self, env):

        import params

        env.set_params(params)

        Logger.info('Checked Analytics-Zoo Status')

        check_process_status(os.path.join(params.zoo_log_dir, params.docker_pid_file))

        Logger.info('Checked Analytics-Zoo Status')

 

if __name__ == "__main__":

  Zoo().execute()

start.sh内容如下

#!/bin/bash

 

systemctl restart docker

if[[ `docker images -q|grep c77c311e82ed` -ne 0 ]];then

docker load -i /tmp/zoo-InsightV4.tar

else

echo "image is exist"

fi

 

zoo_log_dir="/var/log/zoo"

zoo_containerid_file="/var/log/zoo/zoo.containerid"

docker_pid_file="/var/log/zoo/docker.pid"

 

if[ ! -d "$zoo_log_dir" ];then

mkdir -p $zoo_log_dir

fi

 

docker run -itd --net=host zoo-insight-v4:v1.0 --port=$1 --NotebookToken=$2

 

docker ps -q > $zoo_containerid_file

 

pgrep docker -o > $docker_pid_file

stop.sh内容如下

#!/bin/bash

 

docker stop $(docker ps -a -q)

 

docker rm $(docker ps -a -q)

 

zoo_log_dir="/var/log/zoo"

 

zoo_containerid_file="/var/log/zoo/zoo.containerid"

 

docker_pid_file="/var/log/zoo/docker.pid"

 

sleep 3

 

docker ps -q > $zoo_containerid_file

 

systemctl stop docker

 

sleep 5

 

pgrep docker -o > $docker_pid_file

 

    1. 快速链接配置

在quicklinks下创建quicklinks.json文件内容如下

快速链接端口需要与安装时配置端口相同。

 

{

  "name": "default",

  "description": "Analytics-Zoo-webUI default quick links configuration",

  "configuration": {

    "protocol":

    {

      "type":"http"

    },

 

    "links": [

      {

        "name": "Analytics-Zoo-webUI",

        "label": "Analytics-Zoo webUI",

        "requires_user_name": "false",

        "component_name": "ZOO",

        "url":"%@://%@:%@/login",

        "port":{

          "http_default_port": "30111",

          "regex": "^(\\d+)$",

          "site": "analytics_zoo_env"

        }

      }

    ]

  }

}

 

5.重启Ambari安装组件

5.1 清除缓存,重启Ambari

Server1、Server2、Server3:

rm -rf /var/lib/ambari-agent/cache/*

 

重启

Server1:

ambari-server restart

 

Server1、Server2、Server3:

ambari-agent restart

5.2 上传镜像,启动组件

使用工具将zoo-InsightV4.tar上传至manager节点的/tmp文件夹下

在页面执行安装操作即可

重新制作镜像后需要把start.sh中镜像id修改为上传镜像的id

声明:本文内容由网友自发贡献,不代表【wpsshop博客】立场,版权归原作者所有,本站不承担相应法律责任。如您发现有侵权的内容,请联系我们。转载请注明出处:https://www.wpsshop.cn/w/爱喝兽奶帝天荒/article/detail/743267
推荐阅读
相关标签
  

闽ICP备14008679号