关于Cloudera Hadoop的安装（CentOS）- 官文标注（待完成）_centos cloudera hadoop install

In an enterprise data hub, Cloudera Manager and CDH interact with several products such as Apache Accumulo, Apache Impala (incubating), Hue, Cloudera Search, and Cloudera Navigator. This guide provides information about which major and minor release version of a product is supported with which release version of CDH and Cloudera Manager.

Compatibility across different release versions of Cloudera Manager and CDH must be taken into account, especially when carrying out install/upgrade procedures.

当安装、升级时，尤其要考虑Cloudera Manager和CDH版本的兼容性。

JDK compatibility also varies across different Cloudera Manager and CDH versions. Certain versions of CDH 5 are compatible with both JDK 7 and JDK 8. In such cases, ensure all your services are deployed on the same major version. For example, you should not run Hadoop on JDK 7 while running Sqoop on JDK 8. Additionally, since Cloudera does not support mixed environments, all nodes in your cluster must be running the same major JDK version. Cloudera only supports JDKs provided by Oracle.

Cloudera Hadoop集群所有节点必须要使用同一版本的JDK。官方建议安装Oracle JDK，不支持混合版本环境的JDK。

After installing each entity, upgrade to the latest patch version and apply any other appropriate updates. An available update might be specific to the operating system on which it is installed. For example, if you are using CentOS in your environment, you could choose 6 as the major version and 4 as the minor version to indicate that you are using CentOS 6.4. After installing this operating system, apply all relevant CentOS 6.4 upgrades and patches. In some cases, such as some browsers, a minor version might not be listed.

The Cloudera Manager minor version must always be equal to or greater than the CDH minor version. Older versions of Cloudera Manager might not support features in newer versions of CDH. For example, to upgrade to CDH 5.7.1 you must first upgrade to Cloudera Manager 5.7.0.

Cloudera Manager的主版本号和次版本号要和CDH的主次版本号一致或者比CDH的次版本号稍大。比如，CDH为5.7.1那么Cloudera Manager至少为CM5.7.0(CM5.7.1、CM5.7.2都可以使用)

Important: In order to be covered by Cloudera Support:

• All CDH hosts in a logical cluster must run on the same major OS release.                    操作系统主版本号必须一致
• Cloudera temporarily allows a mixed OS configuration during an OS upgrade project.  在操作系统升级时，Cloudera允许混合版本操作系统间的配置。
• Cloudera Manager must run on the same OS release as one of the CDH clusters it manages.CM也必须和CDH安装在相同版本的操作系统上。

Cloudera recommends running the same minor release cross-cluster. However, the risk caused by running different minor OS releases is considered lower than the risk of running different major OS releases.

Gateway hosts may use RHEL/Centos 7.2, subject to some restrictions. See Operating System Support for Gateway Hosts (CDH 5.11 and higher only)

A Gateway host that is dedicated to running Cloudera Data Science Workbench can use RHEL/CentOS 7.2 even if the remaining hosts in your cluster are running any of the other supported operating systems. All hosts must run the same version of the Oracle JDK.

The Hadoop Distributed File System (HDFS) is designed to run on top of an underlying filesystem in an operating system. Cloudera recommends that you use either of the following filesystems tested on thesupported operating systems:

• ext3: This is the most tested underlying filesystem for HDFS.
• ext4: This scalable extension of ext3 is supported in more recent Linux releases.Important: Cloudera does not support in-place upgrades from ext3 to ext4. Cloudera recommends that you format disks as ext4 before using them as data directories.
• XFS: This is the default filesystem in RHEL 7.

Linux filesystems keep metadata that record when each file was accessed. This means that even reads result in a write to the disk. To speed up file reads, Cloudera recommends that you disable this option, called atime, using the mount option in /etc/fstab:

/dev/sdb1 /data1 ext4 defaults,noatime 0

Apply the change without rebooting:

mount -o remount /data1

The filesystem mount options have a sync option that allows you to write synchronously.

However, using the sync option will lead to poor performance for services that write data to disks, such as HDFS, YARN, Kafka and Kudu. In CDH most writes are already replicated. Therefore, having synchronous writes to disks is unnecessary, expensive, and not worth the added safety it provides.

Cloudera Manager and CDH come packaged with an embedded PostgreSQL database, but it is recommended that you configure your cluster with custom external databases, especially in production.

In most cases (but not all), Cloudera supports versions of MariaDB, MySQL and PostgreSQL that are native to each supported Linux distribution.

After installing a database, upgrade to the latest patch and apply appropriate updates. Available updates may be specific to the operating system on which it is installed.

Important: When you restart processes, the configuration for each of the services is redeployed using information saved in the Cloudera Manager database. If this information is not available, your cluster does not start or function correctly. You must schedule and maintain regular backups of the Cloudera Manager database to recover the cluster in the event of the loss of this database.

Only 64 bit JDKs from Oracle are supported. Oracle JDK 7 is supported across all versions of Cloudera Manager 5 and CDH 5. Oracle JDK 8 is supported in C5.3.x and higher.

The following components are supported by the indicated versions of Transport Layer Security (TLS):

The hosts in a Cloudera Manager deployment must satisfy the following networking and security requirements:

• Networking Protocols Support

  CDH requires IPv4. IPv6 is not supported and must be disabled.

  See also Configuring Network Names.

• Multihoming Support

  – Multihoming CDH or Cloudera Manager is not supported outside specifically certified Cloudera partner appliances. Cloudera finds that current Hadoop architectures combined with modern network infrastructures and security practices remove the need for multihoming. Multihoming, however, is beneficial internally in appliance form factors to take advantage of high-bandwidth InfiniBand interconnects.

  Although some subareas of the product may work with unsupported custom multihoming configurations, there are known issues with multihoming. In addition, unknown issues may arise because multihoming is not covered by our test matrix outside the Cloudera-certified partner appliances.

• Entropy

  Data at rest encryption requires sufficient entropy to ensure randomness.

  See Entropy Requirements.

• Cluster hosts must have a working network name resolution system and correctly formatted /etc/hostsfile. All cluster hosts must have properly configured forward and reverse host resolution through DNS. The /etc/hosts files must:
  • Contain consistent information about hostnames and IP addresses across all hosts
  • Not contain uppercase hostnames
  • Not contain duplicate IP addresses

  Cluster hosts must not use aliases, either in /etc/hosts or in configuring DNS. A properly formatted/etc/hosts file should be similar to the following example:

  127.0.0.1 localhost.localdomain localhost
  192.168.1.1 cluster-01.example.com cluster-01
  192.168.1.2 cluster-02.example.com cluster-02
  192.168.1.3 cluster-03.example.com cluster-03 

• In most cases, the Cloudera Manager Server must have SSH access to the cluster hosts when you run the installation or upgrade wizard. You must log in using a root account or an account that has password-less sudo permission. For authentication during the installation and upgrade procedures, you must either enter the password or upload a public and private key pair for the root or sudo user account. If you want to use a public and private key pair, the public key must be installed on the cluster hosts before you use Cloudera Manager.

  Cloudera Manager uses SSH only during the initial install or upgrade. Once the cluster is set up, you can disable root SSH access or change the root password. Cloudera Manager does not save SSH credentials, and all credential information is discarded when the installation is complete.

• If single user mode is not enabled, the Cloudera Manager Agent runs as root so that it can make sure the required directories are created and that processes and files are owned by the appropriate user (for example, the hdfs and mapred users).
• No blocking is done by Security-Enhanced Linux (SELinux).Note: Cloudera Enterprise is supported on platforms with Security-Enhanced Linux (SELinux) enabled. However, Cloudera does not support use of SELinux with Cloudera Navigator. Cloudera is not responsible for policy support nor policy enforcement. If you experience issues with SELinux, contact your OS provider.
• No blocking by iptables or firewalls; port 7180 must be open because it is used to access Cloudera Manager after installation. Cloudera Manager communicates using specific ports, which must be open.
• For RHEL and CentOS, the /etc/sysconfig/network file on each host must contain the hostname you have just set (or verified) for that host.
• Cloudera Manager and CDH use several user accounts and groups to complete their tasks. The set of user accounts and groups varies according to the components you choose to install. Do not delete these accounts or groups and do not modify their permissions and rights. Ensure that no existing systems prevent these accounts and groups from functioning. For example, if you have scripts that delete user accounts not in a whitelist, add these accounts to the list of permitted accounts. Cloudera Manager, CDH, and managed services create and use the following accounts and groups:

This matrix contains compatibility information across versions of Impala and CDH/Cloudera Manager. For detailed information on each release, see the Impala documentation.

Cloudera Distribution of Apache Kafka is currently distributed as a package and in a parcel that is independent of the CDH parcel. The parcel integrates with Cloudera Manager using a Custom Service Descriptor (CSD).

For the latest documentation, see Kafka Documentation.

Apache Kudu is currently distributed as a package and in a parcel that is independent of the CDH parcel. The parcel integrates with Cloudera Manager using a Custom Service Descriptor (CSD).

For the latest documentation, see the Kudu Documentation.

This section describes the CDH and managed service versions supported by the Cloudera Navigator auditing and metadata features.

Cloudera Navigator encryption comprises several components. See Cloudera Navigator Data Encryption Overview for more information on the components, concepts, and architecture involved in encrypting data.

Although the version numbers differ, Cloudera Navigator encryption component releases are generally coordinated with Cloudera Enterprise. See the following table for information on which component versions correspond with a given Cloudera Enterprise release:

Because of a change in the ports used by Key Trustee Server, Navigator Encrypt versions lower than 3.7 and Key Trustee KMS versions lower than 5.4 are not supported in Key Trustee Server 5.4 and higher.

Cloudera Navigator Key HSM must be installed on the same host as Key Trustee Server. Although Key HSM is compatible across all versions of Key Trustee Server, Cloudera strongly recommends also upgrading Key HSM after you upgrade Key Trustee Server. See Installing Cloudera Navigator Key HSM and Upgrading Cloudera Navigator Key HSM for more information.

Sentry enables role-based, fine-grained authorization for HiveServer2 and provides classic database-style authorization for Hive, Cloudera Impala and Cloudera Search. You can use either the Sentry service (introduced in Cloudera Manager 5.1.0 and CDH 5.1.0) or the policy file approach to secure your data.

Note: It's possible for a single cluster to use both the Sentry service (for Hive and Impala) and Sentry policy files (for Solr).

Spark is a fast, general engine for large-scale data processing. For installation and configuration instructions, see Spark Installation. To see new features introduced with each release, refer to the CDH 5 Release Notes.

Since Spark has been integrated into the CDH package, its compatibility with Cloudera Manager and CDH depends on the CDH 5.x.x release it is shipped with.

For compatibility with Cloudera Manager Replication and Snapshot features, see Product Compatibility Matrix for Backup and Disaster Recovery. For information on using Isilon with CDH, see Using CDH with Isilon Storage.

See Using CDH with Isilon Storage.

This section lists supported cloud and virtualization platforms for deploying Cloudera software. The linked reference architectures are not replacements for statements of support above, but are guides to assist with deployment and sizing options.

For information on deploying Cloudera software on a Amazon Web Services (AWS) cloud infrastructure, see the Cloudera Enterprise Reference Architecture for AWS Deployments.

For information on deploying Cloudera software on a Google Cloud Platform infrastructure, see theCloudera Enterprise Reference Architecture for Google Cloud Platform Deployments.

For information on deploying Cloudera software on a Microsoft Azure cloud infrastructure, see theCloudera Enterprise Reference Architecture for Azure Deployments.

For information on deploying Cloudera software on a VMware-based infrastructure, see the Reference architecture for deploying on VMware.

Recommendation when deploying on VMware in the current release: