赞
踩
本文是《spring-cloud-kubernetes实战系列》的第四篇,主要内容是在kubernetes上部署两个应用:Web-Service和Account-Service,通过spring-cloud-kubernetes提供的注册发现能力,实现Web-Service调用Account-Service提供的http服务;
本文由以下段落组成:
本次实战的环境和版本信息如下:
上面的linux、minikube、java、maven,请确保已准备好,linux环境下minikube的安装和启动请参考《Linux安装minikube指南 》。
SpringCloud环境最重要的功能是注册发现服务,因此将SpringCloud应用迁移到kubernetes环境时,开发者最关心的问题是在kubernetes上如何将自身服务暴露出去,以及如何调用其他微服务。
先看看普通SpringCloud环境下的注册发现,下图来自spring官方博客,地址是:https://spring.io/blog/2015/07/14/microservices-with-spring,
由上图可见,应用Account-Service将自己注册到Eureka,这样Web-Service用"account-service"就能在Eureka找到Account-Service服务的地址,然后顺利发送RestFul请求到Account-Service,用上其提供的服务。
如果将上面的Web-Service和Account-Service两个应用迁移到kubernetes上之后,注册发现机制变成了啥样呢?
第一种:沿用上图的方式,将Eureka也部署在kubernetes上,这样的架构和不用kubernetes时没有啥区别;
第二种,就是今天要实战的内容,使用spring-cloud-kubernetes框架,该框架可以调用kubernetes的原生能力来为现有SpringCloud应用提供服务,架构如下图所示:
上图表明,Web-Service应用在调用Account-Service应用的服务时,会用okhttp向API Server请求服务列表,API Server收到请求后会去etcd取数据返回给Web-Service应用,这样Web-Service就有了Account-Service的信息,可以向Account-Service的多个Pod轮询发起请求;
上图有个细节请注意:WebService应用并不是直接将请求发送给Account-Service在kubernetes创建的service,而是直接发送到具体的Pod上了,之所以具有这个能力,是因为spring-cloud-kubernetes框架通过service拿到了Account-Service对应的所有Pod信息(endpoint),此逻辑可以参考源码KubernetesServerList.java,如下所示:
public List<Server> getUpdatedListOfServers() { //用namespace和serviceId做条件,得到该服务对应的所有节点(endpoints)信息 Endpoints endpoints = this.namespace != null ? this.client.endpoints().inNamespace(this.namespace) .withName(this.serviceId).get() : this.client.endpoints().withName(this.serviceId).get(); List<Server> result = new ArrayList<Server>(); if (endpoints != null) { if (LOG.isDebugEnabled()) { LOG.debug("Found [" + endpoints.getSubsets().size() + "] endpoints in namespace [" + this.namespace + "] for name [" + this.serviceId + "] and portName [" + this.portName + "]"); } //遍历所有的endpoint,取出IP地址和端口,构建成Server实例,放入result集合中 for (EndpointSubset subset : endpoints.getSubsets()) { if (subset.getPorts().size() == 1) { EndpointPort port = subset.getPorts().get(FIRST); for (EndpointAddress address : subset.getAddresses()) { result.add(new Server(address.getIp(), port.getPort())); } } else { for (EndpointPort port : subset.getPorts()) { if (Utils.isNullOrEmpty(this.portName) || this.portName.endsWith(port.getName())) { for (EndpointAddress address : subset.getAddresses()) { result.add(new Server(address.getIp(), port.getPort())); } } } } } } else { LOG.warn("Did not find any endpoints in ribbon in namespace [" + this.namespace + "] for name [" + this.serviceId + "] and portName [" + this.portName + "]"); } return result; }
理论分析已经完成,接下来就开始实战吧
如果您不打算写代码,也可以从GitHub上下载本次实战的源码,地址和链接信息如下表所示:
名称 | 链接 | 备注 |
---|---|---|
项目主页 | https://github.com/zq2599/blog_demos | 该项目在GitHub上的主页 |
git仓库地址(https) | https://github.com/zq2599/blog_demos.git | 该项目源码的仓库地址,https协议 |
git仓库地址(ssh) | git@github.com:zq2599/blog_demos.git | 该项目源码的仓库地址,ssh协议 |
这个git项目中有多个文件夹,本章的Account-Service源码在spring-cloud-k8s-account-service文件夹下,Web-Service源码在spring-cloud-k8s-web-service文件夹下,如下图红框所示:
下面是详细的编码过程;
Account-Service服务是个很普通的springboot应用,和spring-cloud-kubernetes没有任何关系:
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>2.1.1.RELEASE</version> <relativePath/> <!-- lookup parent from repository --> </parent> <groupId>com.bolingcavalry</groupId> <artifactId>account-service</artifactId> <version>0.0.1-SNAPSHOT</version> <name>account-service</name> <description>Demo project for Spring Cloud service provider run in kubernetes</description> <properties> <java.version>1.8</java.version> <spring-boot.version>2.1.1.RELEASE</spring-boot.version> <maven-checkstyle-plugin.failsOnError>false</maven-checkstyle-plugin.failsOnError> <maven-checkstyle-plugin.failsOnViolation>false</maven-checkstyle-plugin.failsOnViolation> <maven-checkstyle-plugin.includeTestSourceDirectory>false</maven-checkstyle-plugin.includeTestSourceDirectory> <maven-compiler-plugin.version>3.5</maven-compiler-plugin.version> <maven-deploy-plugin.version>2.8.2</maven-deploy-plugin.version> <maven-failsafe-plugin.version>2.18.1</maven-failsafe-plugin.version> <maven-surefire-plugin.version>2.21.0</maven-surefire-plugin.version> <fabric8.maven.plugin.version>3.5.37</fabric8.maven.plugin.version> <springcloud.version>2.1.1.RELEASE</springcloud.version> </properties> <dependencyManagement> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-dependencies</artifactId> <type>pom</type> <scope>import</scope> <version>${spring-boot.version}</version> </dependency> </dependencies> </dependencyManagement> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter</artifactId> <version>${springcloud.version}</version> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> <version>${springcloud.version}</version> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> <version>${spring-boot.version}</version> <executions> <execution> <goals> <goal>repackage</goal> </goals> </execution> </executions> </plugin> <plugin> <!--skip deploy --> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-deploy-plugin</artifactId> <version>${maven-deploy-plugin.version}</version> <configuration> <skip>true</skip> </configuration> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>${maven-surefire-plugin.version}</version> <configuration> <skipTests>true</skipTests> <!-- Workaround for https://issues.apache.org/jira/browse/SUREFIRE-1588 --> <useSystemClassLoader>false</useSystemClassLoader> </configuration> </plugin> <plugin> <groupId>io.fabric8</groupId> <artifactId>fabric8-maven-plugin</artifactId> <version>${fabric8.maven.plugin.version}</version> <executions> <execution> <id>fmp</id> <goals> <goal>resource</goal> </goals> </execution> </executions> </plugin> </plugins> </build> <profiles> <profile> <id>kubernetes</id> <build> <plugins> <plugin> <groupId>io.fabric8</groupId> <artifactId>fabric8-maven-plugin</artifactId> <version>${fabric8.maven.plugin.version}</version> <executions> <execution> <id>fmp</id> <goals> <goal>resource</goal> <goal>build</goal> </goals> </execution> </executions> <configuration> <enricher> <config> <fmp-service> <type>NodePort</type> </fmp-service> </config> </enricher> </configuration> </plugin> </plugins> </build> </profile> </profiles> </project>
由上面的pom.xml内容可见,account-service应用是个简单的web应用,和SpringCloud、spring-cloud-kubernetes都没有任何关系,和其他springboot唯一的不同就是用到了fabric8-maven-plugin插件,可以方便的将应用部署到kubernetes环境;
spring:
application:
name: account-service
server:
port: 8080
@RestController public class AccountController { private static final Logger LOG = LoggerFactory.getLogger(AccountController.class); private final String hostName = System.getenv("HOSTNAME"); /** * 探针检查响应类 * @return */ @RequestMapping("/health") public String health() { return "OK"; } @RequestMapping("/") public String ribbonPing(){ LOG.info("ribbonPing of {}", hostName); return hostName; } /** * 返回hostname * @return 当前应用所在容器的hostname. */ @RequestMapping("/name") public String getName() { return this.hostName + ", " + new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(new Date()); } }
mvn clean install fabric8:deploy -Dfabric8.generator.from=fabric8/java-jboss-openjdk8-jdk -Pkubernetes
执行成功后控制台输出如下:
... [INFO] Installing /usr/local/work/k8s/ribbon/spring-cloud-k8s-account-service/target/classes/META-INF/fabric8/kubernetes.json to /root/.m2/repository/com/bolingcavalry/account-service/0.0.1-SNAPSHOT/account-service-0.0.1-SNAPSHOT-kubernetes.json [INFO] [INFO] <<< fabric8-maven-plugin:3.5.37:deploy (default-cli) < install @ account-service <<< [INFO] [INFO] [INFO] --- fabric8-maven-plugin:3.5.37:deploy (default-cli) @ account-service --- [INFO] F8: Using Kubernetes at https://192.168.121.133:8443/ in namespace default with manifest /usr/local/work/k8s/ribbon/spring-cloud-k8s-account-service/target/classes/META-INF/fabric8/kubernetes.yml [INFO] Using namespace: default [INFO] Updating a Service from kubernetes.yml [INFO] Updated Service: target/fabric8/applyJson/default/service-account-service.json [INFO] Using namespace: default [INFO] Updating Deployment from kubernetes.yml [INFO] Updated Deployment: target/fabric8/applyJson/default/deployment-account-service.json [INFO] F8: HINT: Use the command `kubectl get pods -w` to watch your pods start up [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 11.941 s [INFO] Finished at: 2019-06-16T19:00:51+08:00 [INFO] ------------------------------------------------------------------------
[root@minikube spring-cloud-k8s-account-service]# kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
account-service 1/1 1 1 69m
[root@minikube spring-cloud-k8s-account-service]# kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
account-service NodePort 10.105.157.201 <none> 8080:32596/TCP 69m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 8d
[root@minikube spring-cloud-k8s-account-service]# minikube service account-service --url
http://192.168.121.133:32596
可见account-service的服务可以通过这个url访问:http://192.168.121.133:32596
Web-Service服务是个springboot应用,用到了spring-cloud-kubernetes提供的注册发现能力,以轮询的方式访问指定服务的全部pod:
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>2.1.1.RELEASE</version> <relativePath/> <!-- lookup parent from repository --> </parent> <groupId>com.bolingcavalry</groupId> <artifactId>web-service</artifactId> <version>0.0.1-SNAPSHOT</version> <name>web-service</name> <description>Demo project for Spring Cloud service consumer run in kubernetes</description> <properties> <java.version>1.8</java.version> <spring-boot.version>2.1.1.RELEASE</spring-boot.version> <maven-checkstyle-plugin.failsOnError>false</maven-checkstyle-plugin.failsOnError> <maven-checkstyle-plugin.failsOnViolation>false</maven-checkstyle-plugin.failsOnViolation> <maven-checkstyle-plugin.includeTestSourceDirectory>false</maven-checkstyle-plugin.includeTestSourceDirectory> <maven-compiler-plugin.version>3.5</maven-compiler-plugin.version> <maven-deploy-plugin.version>2.8.2</maven-deploy-plugin.version> <maven-failsafe-plugin.version>2.18.1</maven-failsafe-plugin.version> <maven-surefire-plugin.version>2.21.0</maven-surefire-plugin.version> <fabric8.maven.plugin.version>3.5.37</fabric8.maven.plugin.version> <springcloud.kubernetes.version>1.0.1.RELEASE</springcloud.kubernetes.version> <springcloud.version>2.1.1.RELEASE</springcloud.version> </properties> <dependencyManagement> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-dependencies</artifactId> <type>pom</type> <scope>import</scope> <version>${spring-boot.version}</version> </dependency> </dependencies> </dependencyManagement> <dependencies> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-kubernetes-core</artifactId> <version>${springcloud.kubernetes.version}</version> </dependency> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-kubernetes-discovery</artifactId> <version>${springcloud.kubernetes.version}</version> </dependency> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-kubernetes-ribbon</artifactId> <version>${springcloud.kubernetes.version}</version> </dependency> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-commons</artifactId> <version>${springcloud.version}</version> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter</artifactId> <version>${springcloud.version}</version> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> <version>${springcloud.version}</version> </dependency> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-netflix-ribbon</artifactId> <version>${springcloud.version}</version> </dependency> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-netflix-hystrix</artifactId> <version>${springcloud.version}</version> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> <version>${spring-boot.version}</version> <executions> <execution> <goals> <goal>repackage</goal> </goals> </execution> </executions> </plugin> <plugin> <!--skip deploy --> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-deploy-plugin</artifactId> <version>${maven-deploy-plugin.version}</version> <configuration> <skip>true</skip> </configuration> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>${maven-surefire-plugin.version}</version> <configuration> <skipTests>true</skipTests> <!-- Workaround for https://issues.apache.org/jira/browse/SUREFIRE-1588 --> <useSystemClassLoader>false</useSystemClassLoader> </configuration> </plugin> <plugin> <groupId>io.fabric8</groupId> <artifactId>fabric8-maven-plugin</artifactId> <version>${fabric8.maven.plugin.version}</version> <executions> <execution> <id>fmp</id> <goals> <goal>resource</goal> </goals> </execution> </executions> </plugin> </plugins> </build> <profiles> <profile> <id>kubernetes</id> <build> <plugins> <plugin> <groupId>io.fabric8</groupId> <artifactId>fabric8-maven-plugin</artifactId> <version>${fabric8.maven.plugin.version}</version> <executions> <execution> <id>fmp</id> <goals> <goal>resource</goal> <goal>build</goal> </goals> </execution> </executions> <configuration> <enricher> <config> <fmp-service> <type>NodePort</type> </fmp-service> </config> </enricher> </configuration> </plugin> </plugins> </build> </profile> </profiles> </project>
spring: application: name: web-service server: port: 8080 backend: ribbon: eureka: enabled: false client: enabled: true ServerListRefreshInterval: 5000 hystrix.command.BackendCall.execution.isolation.thread.timeoutInMilliseconds: 5000 hystrix.threadpool.BackendCallThread.coreSize: 5
package com.bolingcavalry.webservice; import com.netflix.client.config.IClientConfig; import com.netflix.loadbalancer.AvailabilityFilteringRule; import com.netflix.loadbalancer.IPing; import com.netflix.loadbalancer.IRule; import com.netflix.loadbalancer.PingUrl; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.context.annotation.Bean; /** * @Description: ribbon配置类 * @author: willzhao E-mail: zq2599@gmail.com * @date: 2019/6/16 11:52 */ public class RibbonConfiguration { @Autowired IClientConfig ribbonClientConfig; /** * 检查服务是否可用的实例, * 此地址返回的响应的返回码如果是200表示服务可用 * @param config * @return */ @Bean public IPing ribbonPing(IClientConfig config){ return new PingUrl(); } /** * 轮询规则 * @param config * @return */ @Bean public IRule ribbonRule(IClientConfig config){ return new AvailabilityFilteringRule(); } }
package com.bolingcavalry.webservice; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; import org.springframework.cloud.client.circuitbreaker.EnableCircuitBreaker; import org.springframework.cloud.client.discovery.EnableDiscoveryClient; import org.springframework.cloud.client.loadbalancer.LoadBalanced; import org.springframework.cloud.netflix.ribbon.RibbonClient; import org.springframework.context.annotation.Bean; import org.springframework.web.client.RestTemplate; @SpringBootApplication @EnableDiscoveryClient @EnableCircuitBreaker @RibbonClient(name="account-service", configuration = RibbonConfiguration.class) public class WebServiceApplication { public static void main(String[] args) { SpringApplication.run(WebServiceApplication.class, args); } @LoadBalanced @Bean RestTemplate restTemplate(){ return new RestTemplate(); } }
package com.bolingcavalry.webservice; import com.netflix.hystrix.contrib.javanica.annotation.HystrixCommand; import com.netflix.hystrix.contrib.javanica.annotation.HystrixProperty; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Service; import org.springframework.web.client.RestTemplate; import java.text.SimpleDateFormat; import java.util.Date; /** * @Description: 这里面封装了远程调用account-service提供服务的逻辑 * @author: willzhao E-mail: zq2599@gmail.com * @date: 2019/6/16 12:21 */ @Service public class AccountService { @Autowired private RestTemplate restTemplate; @HystrixCommand(fallbackMethod = "getFallbackName" ,commandProperties = { @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "1000") }) public String getDataFromSpringCloudK8SProvider(){ return this.restTemplate.getForObject("http://account-service/name", String.class); } /** * 熔断时调用的方法 * @return */ private String getFallbackName() { return "Fallback" + ", " + new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(new Date()); } }
package com.bolingcavalry.webservice; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.web.bind.annotation.RequestMapping; import org.springframework.web.bind.annotation.RequestParam; import org.springframework.web.bind.annotation.RestController; /** * @Description: 测试用的controller,会远程调用account-service的服务 * @author: willzhao E-mail: zq2599@gmail.com * @date: 2019/6/16 11:46 */ @RestController public class WebServiceController { @Autowired private AccountService accountService; /** * 探针检查响应类 * @return */ @RequestMapping("/health") public String health() { return "OK"; } /** * 远程调用account-service提供的服务 * @return 多次远程调返回的所有结果. */ @RequestMapping("/account") public String account() { StringBuilder sbud = new StringBuilder(); for(int i=0;i<10;i++){ sbud.append(accountService.getDataFromSpringCloudK8SProvider()) .append("<br>"); } return sbud.toString(); } }
mvn clean install fabric8:deploy -Dfabric8.generator.from=fabric8/java-jboss-openjdk8-jdk -Pkubernetes
执行成功后控制台输出如下:
... [INFO] Installing /usr/local/work/k8s/ribbon/spring-cloud-k8s-web-service/target/classes/META-INF/fabric8/kubernetes.json to /root/.m2/repository/com/bolingcavalry/web-service/0.0.1-SNAPSHOT/web-service-0.0.1-SNAPSHOT-kubernetes.json [INFO] [INFO] <<< fabric8-maven-plugin:3.5.37:deploy (default-cli) < install @ web-service <<< [INFO] [INFO] [INFO] --- fabric8-maven-plugin:3.5.37:deploy (default-cli) @ web-service --- [INFO] F8: Using Kubernetes at https://192.168.121.133:8443/ in namespace default with manifest /usr/local/work/k8s/ribbon/spring-cloud-k8s-web-service/target/classes/META-INF/fabric8/kubernetes.yml [INFO] Using namespace: default [INFO] Creating a Service from kubernetes.yml namespace default name web-service [INFO] Created Service: target/fabric8/applyJson/default/service-web-service.json [INFO] Using namespace: default [INFO] Creating a Deployment from kubernetes.yml namespace default name web-service [INFO] Created Deployment: target/fabric8/applyJson/default/deployment-web-service.json [INFO] F8: HINT: Use the command `kubectl get pods -w` to watch your pods start up [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 12.792 s [INFO] Finished at: 2019-06-16T19:24:21+08:00 [INFO] ------------------------------------------------------------------------
[root@minikube spring-cloud-k8s-web-service]# kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
account-service 1/1 1 1 109m
web-service 1/1 1 1 18m
[root@minikube spring-cloud-k8s-web-service]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
account-service NodePort 10.105.157.201 <none> 8080:32596/TCP 109m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 8d
web-service NodePort 10.99.211.179 <none> 8080:30519/TCP 18m
[root@minikube spring-cloud-k8s-web-service]# minikube service web-service --url
http://192.168.121.133:30519
可见web-service的服务可以通过这个url访问:http://192.168.121.133:30519
虽然web-service可以正常调用account-service的服务,但始终访问的是一个pod,接下来我们就对account-service的pod进行扩容,将数量调整为2个,看看web-service是否可以轮询调用每个account-service的pod:
kubectl scale --replicas=2 deployment account-service
[root@minikube spring-cloud-k8s-web-service]# kubectl get pods
NAME READY STATUS RESTARTS AGE
account-service-5554576647-m29xr 1/1 Running 0 53m
account-service-5554576647-zwwml 1/1 Running 0 20s
web-service-6d775855c7-7lkvr 1/1 Running 0 29m
接下来验证web-service配置的熔断服务是否可以生效:
kubectl delete deployment account-service
mvn clean install fabric8:deploy -Dfabric8.generator.from=fabric8/java-jboss-openjdk8-jdk -Pkubernetes
至此,spring-cloud-kubernetes的服务发现和轮询实战(含熔断)就全部完成了,利用API Server提供的信息,spring-cloud-kubernetes将原生的kubernetes服务带给了SpringCloud应用,帮助传统微服务更好的融合在kubernetes环境中,如果您也在考虑将应用迁移到kubernetes上,希望本文能给您一些参考。
Copyright © 2003-2013 www.wpsshop.cn 版权所有,并保留所有权利。