devbox
我家自动化
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

尝试使用官方jailhouse-images仓库运行jailhouse

作者：我家自动化 | 2024-05-19 21:20:18

赞

踩

尝试使用官方jailhouse-images仓库运行jailhouse

jailhouse 的官方 demo 演示仓库

Demo and testing images: https://github.com/siemens/jailhouse-images
通过jailhouse 的官方 demo 演示仓库，可以直接编译出带有部署有jailhouse程序的Linux镜像，有多个目标平台的Linux镜像可选，也有在qemu下的镜像。

参考文档：https://bbs.csdn.net/topics/615164494

运行环境准备

操作系统：Ubuntu22.04
运行主机：PVE 虚拟机

安装git 和 docker

sudo apt install git
sudo apt install docker.io

  • 1
  • 2

配置docker

运行 docker 本身需要 root 权限，而执行脚本时又不允许使用 sudo 命令。所以需要配置 docker 无 sudo 运行。
docker守护程序以root用户身份运行，如果不想以sudo作为docker命令的开头，创建一个名为docker的unix组并将用户添加到其中
创建docker组

sudo groupadd docker

  • 1

把自己的用户添加到docker组

sudo usermod -aG docker $USER

  • 1

激活对组的更改

newgrp docker

  • 1

检测不用sudo能否运行docker指令

docker run hello-world

  • 1

运行记录如下：

scilogyhunter@scilogyhunter-Standard-PC-i440FX-PIIX-1996:~/jailhouse-images$ sudo groupadd docker
[sudo] password for scilogyhunter:
groupadd: group 'docker' already exists
scilogyhunter@scilogyhunter-Standard-PC-i440FX-PIIX-1996:~/jailhouse-images$ sudo usermod -aG docker $USER
scilogyhunter@scilogyhunter-Standard-PC-i440FX-PIIX-1996:~/jailhouse-images$ newgrp docker
scilogyhunter@scilogyhunter-Standard-PC-i440FX-PIIX-1996:~/jailhouse-images$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
c1ec31eb5944: Pull complete
Digest: sha256:a26bff933ddc26d5cdf7faa98b4ae1e3ec20c4985e6f87ac0973052224d24302
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32

下载jailhouse-images

git clone https://github.com/siemens/jailhouse-images.git

  • 1

编译 demo 镜像

进入jailhouse-images 文件夹，执行./kas-container menu 命令。期间要下载 docker 镜像会比较耗时，约 1 个小时。
下载结束后会弹出选择窗口，选择要编译的镜像
在这里插入图片描述

运行过程打印信息。

scilogyhunter@scilogyhunter-Standard-PC-i440FX-PIIX-1996:~/jailhouse-images$ ./kas-container menu
Unable to find image 'ghcr.io/siemens/kas/kas-isar:2.6.2' locally
2.6.2: Pulling from siemens/kas/kas-isar
b380bbd43752: Pull complete
d1bed1a2c34e: Pull complete
560c30a1ba9b: Pull complete
d50a6cc4bf45: Pull complete
34feb3868092: Pull complete
6815c7d4b9d8: Pull complete
3481406e04ed: Pull complete
2c8c2fa9ca93: Pull complete
7065e04db92d: Pull complete
26adc2492566: Pull complete
a060224959b1: Pull complete
Digest: sha256:3ac77685623872c2c6461a7f77ed4694aa8647e44314ed4ad47539078426a904
Status: Downloaded newer image for ghcr.io/siemens/kas/kas-isar:2.6.2
2024-05-08 08:53:53 - INFO     - kas 2.6.2 started
2024-05-08 08:56:15 - INFO     - Saving configuration as /work/.config.yaml
2024-05-08 08:56:15 - INFO     - Saving configuration as /work/.config.yaml
2024-05-08 08:56:15 - INFO     - /repo$ git rev-parse --show-toplevel
2024-05-08 08:56:15 - INFO     - Using /repo as root for repository jailhouse
2024-05-08 08:56:15 - INFO     - /work$ git clone -q https://github.com/ilbers/isar /work/isar
2024-05-08 08:57:33 - INFO     - Repository isar cloned
2024-05-08 08:57:33 - INFO     - /repo$ git rev-parse --show-toplevel
2024-05-08 08:57:33 - INFO     - Using /repo as root for repository jailhouse
2024-05-08 08:57:33 - INFO     - /work/isar$ git status -s
2024-05-08 08:57:33 - INFO     - /work/isar$ git rev-parse --verify -q origin/5f87d48f279761c2fefef1db16b4ccb5b955a364
2024-05-08 08:57:33 - INFO     - /work/isar$ git checkout -q 5f87d48f279761c2fefef1db16b4ccb5b955a364
2024-05-08 08:57:33 - INFO     - /repo$ git rev-parse --show-toplevel
2024-05-08 08:57:33 - INFO     - Using /repo as root for repository jailhouse
2024-05-08 08:57:33 - INFO     - /repo$ git rev-parse --show-toplevel
2024-05-08 08:57:33 - INFO     - Using /repo as root for repository jailhouse
2024-05-08 08:57:33 - INFO     - /work/isar$ /tmp/tmpodpia85r/get_bb_env /build
2024-05-08 08:57:33 - INFO     - /repo$ git rev-parse --show-toplevel
2024-05-08 08:57:33 - INFO     - Using /repo as root for repository jailhouse
2024-05-08 08:57:33 - INFO     - /build$ /work/isar/bitbake/bin/bitbake -c build mc:qemu-amd64-jailhouse-demo:demo-image
Parsing recipes: 100% |######################################################################################################################################| Time: 0:00:00
Parsing of 38 .bb files complete (0 cached, 38 parsed). 76 targets, 0 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
Initialising tasks: 100% |###################################################################################################################################| Time: 0:00:00
NOTE: No setscene tasks
NOTE: Executing Tasks
WARNING: mc:qemu-amd64-jailhouse-demo:linux-jailhouse-5.15.86-r0 do_fetch: Failed to fetch URL https://github.com/siemens/linux/archive/9f943ff23cec02b0b2db9e777197fb298ca2c4aa.tar.gz;downloadfilename=linux-9f943ff23cec02b0b2db9e777197fb298ca2c4aa.tar.gz, attempting MIRRORS if available
ERROR: mc:qemu-amd64-jailhouse-demo:linux-jailhouse-5.15.86-r0 do_fetch: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; unset _PYTHON_SYSCONFIGDATA_NAME; export GIT_PROXY_COMMAND="oe-git-proxy"; export NO_PROXY="*"; export PATH="/work/isar/scripts:/work/isar/bitbake/bin:/usr/sbin:/usr/bin:/sbin:/bin"; export HOME="/tmp/tmpjyf8u5av"; /usr/bin/env wget -t 2 -T 30 --passive-ftp --no-check-certificate -O /build/downloads/linux-9f943ff23cec02b0b2db9e777197fb298ca2c4aa.tar.gz -P /build/downloads 'https://github.com/siemens/linux/archive/9f943ff23cec02b0b2db9e777197fb298ca2c4aa.tar.gz' --progress=dot -v failed with exit code 4, output:
--2024-05-08 08:57:35--  https://github.com/siemens/linux/archive/9f943ff23cec02b0b2db9e777197fb298ca2c4aa.tar.gz
Resolving github.com (github.com)... 20.205.243.166
Connecting to github.com (github.com)|20.205.243.166|:443... connected.
HTTP request sent, awaiting response... Read error (Success.) in headers.
Retrying.

--2024-05-08 08:57:46--  (try: 2)  https://github.com/siemens/linux/archive/9f943ff23cec02b0b2db9e777197fb298ca2c4aa.tar.gz
Connecting to github.com (github.com)|20.205.243.166|:443... failed: Connection timed out.
Resolving github.com (github.com)... 20.205.243.166
Connecting to github.com (github.com)|20.205.243.166|:443... failed: Connection timed out.
Giving up.


ERROR: mc:qemu-amd64-jailhouse-demo:linux-jailhouse-5.15.86-r0 do_fetch: Fetcher failure for URL: 'https://github.com/siemens/linux/archive/9f943ff23cec02b0b2db9e777197fb298ca2c4aa.tar.gz;downloadfilename=linux-9f943ff23cec02b0b2db9e777197fb298ca2c4aa.tar.gz'. Unable to fetch URL from any source.
ERROR: Logfile of failure stored in: /build/tmp/work/jailhouse-demo-amd64/linux-jailhouse/5.15.86-r0/temp/log.do_fetch.255
ERROR: Task (mc:qemu-amd64-jailhouse-demo:/repo/recipes-kernel/linux/linux-jailhouse_5.15.86.bb:do_fetch) failed with exit code '1'
ERROR: mc:qemu-amd64-jailhouse-demo:jailhouse-jailhouse-0.9999-next-r0 do_fetch: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; unset _PYTHON_SYSCONFIGDATA_NAME; export GIT_PROXY_COMMAND="oe-git-proxy"; export NO_PROXY="*"; export PATH="/work/isar/scripts:/work/isar/bitbake/bin:/usr/sbin:/usr/bin:/sbin:/bin"; export HOME="/tmp/tmpjyf8u5av"; git -c core.fsyncobjectfiles=0 ls-remote https://github.com/siemens/jailhouse  failed with exit code 128, output:
fatal: unable to access 'https://github.com/siemens/jailhouse/': Operation timed out after 300028 milliseconds with 0 out of 0 bytes received

ERROR: Logfile of failure stored in: /build/tmp/work/jailhouse-demo-amd64/jailhouse-jailhouse/0.9999-next-r0/temp/log.do_fetch.305
ERROR: Task (mc:qemu-amd64-jailhouse-demo:/repo/recipes-jailhouse/jailhouse/jailhouse_latest.bb:do_fetch) failed with exit code '1'
NOTE: Tasks Summary: Attempted 21 tasks of which 0 didn't need to be rerun and 2 failed.

Summary: 2 tasks failed:
  mc:qemu-amd64-jailhouse-demo:/repo/recipes-kernel/linux/linux-jailhouse_5.15.86.bb:do_fetch
  mc:qemu-amd64-jailhouse-demo:/repo/recipes-jailhouse/jailhouse/jailhouse_latest.bb:do_fetch
Summary: There was 1 WARNING message shown.
Summary: There were 3 ERROR messages shown, returning a non-zero exit code.
2024-05-08 09:04:02 - ERROR    - Command returned non-zero exit status 1
scilogyhunter@scilogyhunter-Standard-PC-i440FX-PIIX-1996:~/jailhouse-images$

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54
  • 55
  • 56
  • 57
  • 58
  • 59
  • 60
  • 61
  • 62
  • 63
  • 64
  • 65
  • 66
  • 67
  • 68
  • 69
  • 70
  • 71
  • 72
  • 73
  • 74
  • 75
  • 76
  • 77

运行报错，下载
https://github.com/siemens/linux/archive/9f943ff23cec02b0b2db9e777197fb298ca2c4aa.tar.gz;downloadfilename=linux-9f943ff23cec02b0b2db9e777197fb298ca2c4aa.tar.gz失败。

ERROR: mc:qemu-amd64-jailhouse-demo:linux-jailhouse-5.15.86-r0 do_fetch: Fetcher failure for URL: 'https://github.com/siemens/linux/archive/9f943ff23cec02b0b2db9e777197fb298ca2c4aa.tar.gz;downloadfilename=linux-9f943ff23cec02b0b2db9e777197fb298ca2c4aa.tar.gz'. Unable to fetch URL from any source.

  • 1

尝试再次运行，编译成功。
之前的报错可能是因为需要境外下载，速度太慢，总是超时的原因。可以考虑翻墙加速。

scilogyhunter@scilogyhunter-Standard-PC-i440FX-PIIX-1996:~/jailhouse-images$ ./kas-container menu
2024-05-08 09:24:13 - INFO     - kas 2.6.2 started
2024-05-08 09:27:13 - INFO     - /repo$ git rev-parse --show-toplevel
2024-05-08 09:27:13 - INFO     - Using /repo as root for repository jailhouse
2024-05-08 09:27:13 - INFO     - /work/isar$ git remote set-url origin https://github.com/ilbers/isar
2024-05-08 09:27:13 - INFO     - /work/isar$ git cat-file -t 5f87d48f279761c2fefef1db16b4ccb5b955a364
2024-05-08 09:27:13 - INFO     - Repository isar already contains 5f87d48f279761c2fefef1db16b4ccb5b955a364 as commit
2024-05-08 09:27:13 - INFO     - /repo$ git rev-parse --show-toplevel
2024-05-08 09:27:13 - INFO     - Using /repo as root for repository jailhouse
2024-05-08 09:27:13 - INFO     - /work/isar$ git status -s
2024-05-08 09:27:14 - INFO     - /work/isar$ git rev-parse --verify -q origin/5f87d48f279761c2fefef1db16b4ccb5b955a364
2024-05-08 09:27:14 - INFO     - /work/isar$ git checkout -q 5f87d48f279761c2fefef1db16b4ccb5b955a364
2024-05-08 09:27:14 - INFO     - /repo$ git rev-parse --show-toplevel
2024-05-08 09:27:14 - INFO     - Using /repo as root for repository jailhouse
2024-05-08 09:27:14 - INFO     - /repo$ git rev-parse --show-toplevel
2024-05-08 09:27:14 - INFO     - Using /repo as root for repository jailhouse
2024-05-08 09:27:14 - INFO     - /work/isar$ /tmp/tmpitjgwfjl/get_bb_env /build
2024-05-08 09:27:14 - INFO     - /repo$ git rev-parse --show-toplevel
2024-05-08 09:27:14 - INFO     - Using /repo as root for repository jailhouse
2024-05-08 09:27:14 - INFO     - /build$ /work/isar/bitbake/bin/bitbake -c build mc:qemu-amd64-jailhouse-demo:demo-image
Loading cache: 100% |########################################################################################################################################| Time: 0:00:00
Loaded 76 entries from dependency cache.
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
Initialising tasks: 100% |###################################################################################################################################| Time: 0:00:00
NOTE: No setscene tasks
NOTE: Executing Tasks
jailhouse-latest: Building revision d6e64f901e6d
NOTE: Tasks Summary: Attempted 89 tasks of which 19 didn't need to be rerun and all succeeded.
scilogyhunter@scilogyhunter-Standard-PC-i440FX-PIIX-1996:~/jailhouse-images$

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32

编译出的镜像在build/tmp/deploy/images/qemu-amd64/路径下

scilogyhunter@scilogyhunter-Standard-PC-i440FX-PIIX-1996:~/jailhouse-images$ ls build/tmp/deploy/images/qemu-amd64/
demo-image-1.0-r0.dpkg_status  demo-image-jailhouse-demo-qemu-amd64.ext4.img    demo-image-jailhouse-demo-qemu-amd64-vmlinuz
demo-image-1.0-r0.manifest     demo-image-jailhouse-demo-qemu-amd64-initrd.img
scilogyhunter@scilogyhunter-Standard-PC-i440FX-PIIX-1996:~/jailhouse-images$

  • 1
  • 2
  • 3
  • 4

安装 qemu

scilogyhunter@scilogyhunter-Standard-PC-i440FX-PIIX-1996:~/jailhouse-images$ sudo apt install qemu qemu-kvm virt-manager bridge-utils
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'qemu-system-x86' instead of 'qemu-kvm'
。
。

scilogyhunter@scilogyhunter-Standard-PC-i440FX-PIIX-1996:~/jailhouse-images$ qemu-system-x86_64 -version
QEMU emulator version 6.2.0 (Debian 1:6.2+dfsg-2ubuntu6.19)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers
scilogyhunter@scilogyhunter-Standard-PC-i440FX-PIIX-1996:~/jailhouse-images$ uname -a
Linux scilogyhunter-Standard-PC-i440FX-PIIX-1996 6.5.0-27-generic #28~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Mar 15 10:51:06 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
scilogyhunter@scilogyhunter-Standard-PC-i440FX-PIIX-1996:~/jailhouse-images$

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14

在qemu 中运行 jailhouse 镜像

执行./start-qemu.sh x86 命令，让编译出的 x86-qemu-jailhouse 镜像在 qemu 中运行。
在这里插入图片描述

执行命令行，命令行会停止，在图形界面是上会出现启动的 qemu 系统窗口。
登录系统，用户名密码都是 root
在这里插入图片描述

接下来在qemu 系统窗口中运行 jailhouse。

jailhouse 运行

  1. 安装 内核模块
sudo  insmod  /lib/modules/5.10.153-phytium-embeded-v2.0/jailhouse/driver/jailhouse.ko 
sudo  insmod  /lib/modules/4.19.246-phytium-embeded-v2.0/jailhouse/driver/jailhouse.ko

  • 1
  • 2
  1. 启动 jailhouse
sudo jailhouse enable /etc/jailhouse/e2000q.cell

  • 1
  1. 查看 cell 列表
jailhouse  cell list

  • 1
  1. 创建非 root cell
sudo jailhouse cell create /etc/jailhouse/e2000q-inmate-demo.cell
sudo jailhouse cell create /etc/jailhouse/e2000q-inmate-rtos64.cell;

  • 1
  • 2
  1. 加载非 root cell 镜像
sudo jailhouse cell load 1 /usr/local/libexec/jailhouse/demos/gic-demo.bin
sudo jailhouse cell load 1 /usr/local/libexec/jailhouse/demos/ivshmem-demo.bin
sudo jailhouse cell load 1 /usr/local/libexec/jailhouse/demos/uart-demo.bin
sudo jailhouse cell load 1 freertos64.bin --address 0x80100000;

  • 1
  • 2
  • 3
  • 4
  1. 启动非 root cell
sudo jailhouse cell start 1;

  • 1

当前运行环境是 inter x86 安装 PVE，PVE 下安装 Ubuntu2204 虚拟机，Ubuntu2204 虚拟机下安装qemu,qemu 里跑编译出的 jailhouse 系统镜像，启动jailhouse 系统会卡死。

下面换到inter x86 直接安装Ubuntu2204，Ubuntu2204 下安装qemu,qemu 里跑编译出的 jailhouse 系统镜像。执行效果和在 PVE 虚拟机里的情况一样，在使能 jailhouse 时，完全卡死。

scilogyhunter@ubuntu2204:~/jailhouse-images$ ./start-qemu.sh x86

Initializing Jailhouse hypervisor v0.12 (342-gd6e64f90) on CPU 1
Code location: 0xfffffffff0000050
Using x2APIC
Page pool usage after early setup: mem 47/975, remap 0/131072
Initializing processors:
 CPU 1... (APIC ID 1) OK
 CPU 3... (APIC ID 3) OK
 CPU 0... (APIC ID 0) OK
 CPU 2... (APIC ID 2) OK
Initializing unit: VT-d
DMAR unit @0xfed90000/0x1000
Reserving 24 interrupt(s) for device ff:00.0 at index 0
Initializing unit: IOAPIC
Initializing unit: Cache Allocation Technology
Initializing unit: PCI
Adding PCI device 00:01.0 to cell "QEMU-VM"
Adding PCI device 00:02.0 to cell "QEMU-VM"
Reserving 5 interrupt(s) for device 00:02.0 at index 24
Adding PCI device 00:03.0 to cell "QEMU-VM"
Reserving 1 interrupt(s) for device 00:03.0 at index 29
Adding PCI device 00:1b.0 to cell "QEMU-VM"
Reserving 1 interrupt(s) for device 00:1b.0 at index 30
Adding PCI device 00:1f.0 to cell "QEMU-VM"
Adding PCI device 00:1f.2 to cell "QEMU-VM"
Reserving 1 interrupt(s) for device 00:1f.2 at index 31
Adding PCI device 00:1f.3 to cell "QEMU-VM"
Adding PCI device 00:1f.7 to cell "QEMU-VM"
Reserving 2 interrupt(s) for device 00:1f.7 at index 32
Adding virtual PCI device 01:0c.0 to cell "QEMU-VM"
Adding virtual PCI device 01:0d.0 to cell "QEMU-VM"
Adding virtual PCI device 01:0e.0 to cell "QEMU-VM"
Adding virtual PCI device 01:0f.0 to cell "QEMU-VM"
Page pool usage after late setup: mem 271/975, remap 65543/131072
Activating hypervisor
FATAL: Invalid MMIO/RAM read, addr: 0x00000000fead4024 size: 4
RIP: 0xffffffffa01a37ab RSP: 0xffffc90000124ef8 FLAGS: 10046
RAX: 0xffffc900003dc000 RBX: 0xffff8880033f4828 RCX: 0x000000000000080b
RDX: 0x0000000000000001 RSI: 0xffff8880033f4828 RDI: 0xffff8880033f4cc0
CS: 10 BASE: 0x0000000000000000 AR-BYTES: a09b EFER.LMA 1
CR0: 0x0000000080050033 CR3: 0x00000000052ce005 CR4: 0x0000000000372ea0
EFER: 0x0000000000000d01
Parking CPU 3 (Cell: "QEMU-VM")
FATAL: Invalid MMIO/RAM read, addr: 0x00000000fea84000 size: 4
RIP: 0xffffffffa00107af RSP: 0xffffc90000143e18 FLAGS: 10246
RAX: 0xffffc90000360000 RBX: 0xffff88803f3c88c0 RCX: 0x0001020304050608
RDX: 0x0000000000000001 RSI: 0x0000000000000000 RDI: 0xffff88803f3c88c0
CS: 10 BASE: 0x0000000000000000 AR-BYTES: a09b EFER.LMA 1
CR0: 0x0000000080050033 CR3: 0x000000000200a003 CR4: 0x0000000000372eb0
EFER: 0x0000000000000d01
Parking CPU 0 (Cell: "QEMU-VM")
FATAL: Invalid MMIO/RAM write, addr: 0x00000000feada3b8 size: 4
RIP: 0xffffffff815a294e RSP: 0xffffc90000333bc8 FLAGS: 10006
RAX: 0xffffc90000039380 RBX: 0xffff8880037fc8b0 RCX: 0x0000000000000008
RDX: 0x0000000000000100 RSI: 0xffff8880050ac028 RDI: 0xffff8880037fc000
CS: 10 BASE: 0x0000000000000000 AR-BYTES: a09b EFER.LMA 1
CR0: 0x0000000080050033 CR3: 0x0000000005008005 CR4: 0x0000000000372ea0
EFER: 0x0000000000000d01
Parking CPU 1 (Cell: "QEMU-VM")

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54
  • 55
  • 56
  • 57
  • 58
  • 59
  • 60

尝试编译一个 x86 物理机里运行的 jailhouse 镜像，然后安装到 x86 物理机上运行。
在这里插入图片描述

编译报错

scilogyhunter@scilogyhunter-Standard-PC-i440FX-PIIX-1996:~/jailhouse-images$ ./kas-container menu
2024-05-10 07:43:58 - INFO     - kas 2.6.2 started
2024-05-10 07:44:15 - INFO     - /repo$ git rev-parse --show-toplevel
2024-05-10 07:44:15 - INFO     - Using /repo as root for repository jailhouse
2024-05-10 07:44:15 - INFO     - /work/isar$ git remote set-url origin https://github.com/ilbers/isar
2024-05-10 07:44:15 - INFO     - /work/isar$ git cat-file -t 5f87d48f279761c2fefef1db16b4ccb5b955a364
2024-05-10 07:44:15 - INFO     - Repository isar already contains 5f87d48f279761c2fefef1db16b4ccb5b955a364 as commit
2024-05-10 07:44:15 - INFO     - /repo$ git rev-parse --show-toplevel
2024-05-10 07:44:15 - INFO     - Using /repo as root for repository jailhouse
2024-05-10 07:44:15 - INFO     - /work/isar$ git status -s
2024-05-10 07:44:15 - INFO     - /work/isar$ git rev-parse --verify -q origin/5f87d48f279761c2fefef1db16b4ccb5b955a364
2024-05-10 07:44:15 - INFO     - /work/isar$ git checkout -q 5f87d48f279761c2fefef1db16b4ccb5b955a364
2024-05-10 07:44:15 - INFO     - /repo$ git rev-parse --show-toplevel
2024-05-10 07:44:15 - INFO     - Using /repo as root for repository jailhouse
2024-05-10 07:44:15 - INFO     - /repo$ git rev-parse --show-toplevel
2024-05-10 07:44:15 - INFO     - Using /repo as root for repository jailhouse
2024-05-10 07:44:15 - INFO     - /work/isar$ /tmp/tmpt2naj7ei/get_bb_env /build
2024-05-10 07:44:15 - INFO     - /repo$ git rev-parse --show-toplevel
2024-05-10 07:44:15 - INFO     - Using /repo as root for repository jailhouse
2024-05-10 07:44:15 - INFO     - /build$ /work/isar/bitbake/bin/bitbake -c build mc:nuc6cay-jailhouse-demo:demo-image
Loading cache: 100% |########################################################################################################################################| Time: 0:00:00
Loaded 76 entries from dependency cache.
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
Initialising tasks: 100% |###################################################################################################################################| Time: 0:00:00
NOTE: No setscene tasks
NOTE: Executing Tasks
ERROR: mc:nuc6cay-jailhouse-demo:expand-on-first-boot-1.1-r0 do_dpkg_build: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:do_dpkg_build(d)
     0003:
File: '/work/isar/meta/classes/dpkg-base.bbclass', lineno: 209, function: do_dpkg_build
     0205:    lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
     0206:                             shared=True)
     0207:    bb.build.exec_func("dpkg_do_mounts", d)
     0208:    try:
 *** 0209:        bb.build.exec_func("dpkg_runbuild", d)
     0210:    finally:
     0211:        bb.build.exec_func("dpkg_undo_mounts", d)
     0212:        bb.utils.unlockfile(lock)
     0213:}
File: '/work/isar/bitbake/lib/bb/build.py', lineno: 251, function: exec_func
     0247:    with bb.utils.fileslocked(lockfiles):
     0248:        if ispython:
     0249:            exec_func_python(func, d, runfile, cwd=adir)
     0250:        else:
 *** 0251:            exec_func_shell(func, d, runfile, cwd=adir)
     0252:
     0253:    try:
     0254:        curcwd = os.getcwd()
     0255:    except:
File: '/work/isar/bitbake/lib/bb/build.py', lineno: 452, function: exec_func_shell
     0448:    with open(fifopath, 'r+b', buffering=0) as fifo:
     0449:        try:
     0450:            bb.debug(2, "Executing shell function %s" % func)
     0451:            with open(os.devnull, 'r+') as stdin, logfile:
 *** 0452:                bb.process.run(cmd, shell=False, stdin=stdin, log=logfile, extrafiles=[(fifo,readfifo)])
     0453:        finally:
     0454:            os.unlink(fifopath)
     0455:
     0456:    bb.debug(2, "Shell function %s finished" % func)
File: '/work/isar/bitbake/lib/bb/process.py', lineno: 182, function: run
     0178:        if not stderr is None:
     0179:            stderr = stderr.decode("utf-8")
     0180:
     0181:    if pipe.returncode != 0:
 *** 0182:        raise ExecutionError(cmd, pipe.returncode, stdout, stderr)
     0183:    return stdout, stderr
Exception: bb.process.ExecutionError: Execution of '/build/tmp/work/jailhouse-demo-amd64/expand-on-first-boot/1.1-r0/temp/run.dpkg_runbuild.72' failed with exit code 1:
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
This script can only be run as root:root or builder:<gid of builder>!
id: cannot find name for group ID 1000
(Currently running as builder(1000):1000(1000))
WARNING: exit code 1 from a shell command.


ERROR: Logfile of failure stored in: /build/tmp/work/jailhouse-demo-amd64/expand-on-first-boot/1.1-r0/temp/log.do_dpkg_build.72
ERROR: mc:nuc6cay-jailhouse-demo:customizations-nuc6cay-1.0-r0 do_dpkg_build: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:do_dpkg_build(d)
     0003:
File: '/work/isar/meta/classes/dpkg-base.bbclass', lineno: 209, function: do_dpkg_build
     0205:    lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
     0206:                             shared=True)
     0207:    bb.build.exec_func("dpkg_do_mounts", d)
     0208:    try:
 *** 0209:        bb.build.exec_func("dpkg_runbuild", d)
     0210:    finally:
     0211:        bb.build.exec_func("dpkg_undo_mounts", d)
     0212:        bb.utils.unlockfile(lock)
     0213:}
File: '/work/isar/bitbake/lib/bb/build.py', lineno: 251, function: exec_func
     0247:    with bb.utils.fileslocked(lockfiles):
     0248:        if ispython:
     0249:            exec_func_python(func, d, runfile, cwd=adir)
     0250:        else:
 *** 0251:            exec_func_shell(func, d, runfile, cwd=adir)
     0252:
     0253:    try:
     0254:        curcwd = os.getcwd()
     0255:    except:
File: '/work/isar/bitbake/lib/bb/build.py', lineno: 452, function: exec_func_shell
     0448:    with open(fifopath, 'r+b', buffering=0) as fifo:
     0449:        try:
     0450:            bb.debug(2, "Executing shell function %s" % func)
     0451:            with open(os.devnull, 'r+') as stdin, logfile:
 *** 0452:                bb.process.run(cmd, shell=False, stdin=stdin, log=logfile, extrafiles=[(fifo,readfifo)])
     0453:        finally:
     0454:            os.unlink(fifopath)
     0455:
     0456:    bb.debug(2, "Shell function %s finished" % func)
File: '/work/isar/bitbake/lib/bb/process.py', lineno: 182, function: run
     0178:        if not stderr is None:
     0179:            stderr = stderr.decode("utf-8")
     0180:
     0181:    if pipe.returncode != 0:
 *** 0182:        raise ExecutionError(cmd, pipe.returncode, stdout, stderr)
     0183:    return stdout, stderr
Exception: bb.process.ExecutionError: Execution of '/build/tmp/work/jailhouse-demo-amd64/customizations-nuc6cay/1.0-r0/temp/run.dpkg_runbuild.71' failed with exit code 1:
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
This script can only be run as root:root or builder:<gid of builder>!
id: cannot find name for group ID 1000
(Currently running as builder(1000):1000(1000))
WARNING: exit code 1 from a shell command.


ERROR: Logfile of failure stored in: /build/tmp/work/jailhouse-demo-amd64/customizations-nuc6cay/1.0-r0/temp/log.do_dpkg_build.71
ERROR: Task (mc:nuc6cay-jailhouse-demo:/work/isar/meta/recipes-support/expand-on-first-boot/expand-on-first-boot_1.1.bb:do_dpkg_build) failed with exit code '1'
ERROR: Task (mc:nuc6cay-jailhouse-demo:/repo/recipes-core/customizations/customizations.bb:do_dpkg_build) failed with exit code '1'
NOTE: Tasks Summary: Attempted 88 tasks of which 86 didn't need to be rerun and 2 failed.

Summary: 2 tasks failed:
  mc:nuc6cay-jailhouse-demo:/work/isar/meta/recipes-support/expand-on-first-boot/expand-on-first-boot_1.1.bb:do_dpkg_build
  mc:nuc6cay-jailhouse-demo:/repo/recipes-core/customizations/customizations.bb:do_dpkg_build
Summary: There were 2 ERROR messages shown, returning a non-zero exit code.
2024-05-10 07:44:18 - ERROR    - Command returned non-zero exit status 1
scilogyhunter@scilogyhunter-Standard-PC-i440FX-PIIX-1996:~/jailhouse-images$

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54
  • 55
  • 56
  • 57
  • 58
  • 59
  • 60
  • 61
  • 62
  • 63
  • 64
  • 65
  • 66
  • 67
  • 68
  • 69
  • 70
  • 71
  • 72
  • 73
  • 74
  • 75
  • 76
  • 77
  • 78
  • 79
  • 80
  • 81
  • 82
  • 83
  • 84
  • 85
  • 86
  • 87
  • 88
  • 89
  • 90
  • 91
  • 92
  • 93
  • 94
  • 95
  • 96
  • 97
  • 98
  • 99
  • 100
  • 101
  • 102
  • 103
  • 104
  • 105
  • 106
  • 107
  • 108
  • 109
  • 110
  • 111
  • 112
  • 113
  • 114
  • 115
  • 116
  • 117
  • 118
  • 119
  • 120
  • 121
  • 122
  • 123
  • 124
  • 125
  • 126
  • 127
  • 128
  • 129
  • 130
  • 131
  • 132
  • 133
  • 134
  • 135
  • 136
  • 137
  • 138
  • 139
  • 140
  • 141
  • 142
  • 143
  • 144
  • 145

尝试 v0.12 版 jailhouse-images

git clone -b v0.12 https://github.com/siemens/jailhouse-images.git
scilogyhunter@scilogyhunter-Standard-PC-i440FX-PIIX-1996:~$ mkdir jailhouse-images-v0.12
scilogyhunter@scilogyhunter-Standard-PC-i440FX-PIIX-1996:~$ ls
Desktop  Documents  Downloads  jailhouse-images  jailhouse-images-v0.12  Music  Pictures  Public  snap  Templates  Videos
scilogyhunter@scilogyhunter-Standard-PC-i440FX-PIIX-1996:~$ cd jailhouse-images-v0.12/
scilogyhunter@scilogyhunter-Standard-PC-i440FX-PIIX-1996:~/jailhouse-images-v0.12$ git clone -b v0.12 https://github.com/siemens/jailhouse-images.git
Cloning into 'jailhouse-images'...
remote: Enumerating objects: 2545, done.
remote: Counting objects: 100% (231/231), done.
remote: Compressing objects: 100% (40/40), done.
remote: Total 2545 (delta 203), reused 191 (delta 191), pack-reused 2314
Receiving objects: 100% (2545/2545), 808.10 KiB | 156.00 KiB/s, done.
Resolving deltas: 100% (1393/1393), done.
Note: switching to '9e57d07c806a4e80ddd0913d894c1c54bb9223fe'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

scilogyhunter@scilogyhunter-Standard-PC-i440FX-PIIX-1996:~/jailhouse-images-v0.12$ ls
jailhouse-images
scilogyhunter@scilogyhunter-Standard-PC-i440FX-PIIX-1996:~/jailhouse-images-v0.12$ cd jailhouse-images/
scilogyhunter@scilogyhunter-Standard-PC-i440FX-PIIX-1996:~/jailhouse-images-v0.12/jailhouse-images$ ls
build-images.sh  CONTRIBUTING.md  images.list  opt-latest.yml  README.md    recipes-core      recipes-jailhouse  scripts        wic
conf             COPYING          kas.yml      opt-rt.yml      recipes-bsp  recipes-devtools  recipes-kernel     start-qemu.sh
scilogyhunter@scilogyhunter-Standard-PC-i440FX-PIIX-1996:~/jailhouse-images-v0.12/jailhouse-images$
这里运行 ./build-images.sh 命令
scilogyhunter@scilogyhunter-Standard-PC-i440FX-PIIX-1996:~/jailhouse-images-v0.12/jailhouse-images$ ./build-images.sh
Available images demo images:
 1: QEMU/KVM Intel-x86 virtual target
 2: QEMU ARM64 virtual target
 3: Orange Pi Zero (256 MB edition)
 4: Intel NUC (NUC6CAY, 8 GB RAM)
 5: Marvell ESPRESSObin (1 GB edition)
 6: Marvell MACCHIATObin
 7: LeMaker HiKey (Kirin 620 SoC, 2 GB edition)
 8: Avnet Ultra96
 9: Raspberry Pi 4 (1 GB edition)
 0: all (may take hours...)

Select images to build (space-separated index list):

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52

最后运行还是报错。

scilogyhunter@scilogyhunter-Standard-PC-i440FX-PIIX-1996:~/jailhouse-images-v0.12/jailhouse-images$ ./build-images.sh
Available images demo images:
 1: QEMU/KVM Intel-x86 virtual target
 2: QEMU ARM64 virtual target
 3: Orange Pi Zero (256 MB edition)
 4: Intel NUC (NUC6CAY, 8 GB RAM)
 5: Marvell ESPRESSObin (1 GB edition)
 6: Marvell MACCHIATObin
 7: LeMaker HiKey (Kirin 620 SoC, 2 GB edition)
 8: Avnet Ultra96
 9: Raspberry Pi 4 (1 GB edition)
 0: all (may take hours...)

Select images to build (space-separated index list): 1
./kas-docker                               100%[========================================================================================>]   6.36K  15.2KB/s    in 0.4s
Unable to find image 'kasproject/kas-isar:2.0' locally
2.0: Pulling from kasproject/kas-isar
8d691f585fa8: Pull complete
62ad463f507f: Pull complete
920ec0e0896c: Pull complete
006bc8e40609: Pull complete                                                                                                                                                 86ce1358af8a: Pull complete
fc5c466deaad: Pull complete
25ffe6e5147c: Pull complete
aa6087619282: Pull complete
a1f37ac00437: Pull complete                                                                                                                                                 3cb4a1d166c8: Pull complete
8f349537ba7d: Pull complete
Digest: sha256:ccfe3090d86746d94753f9a07fb23770355e51f1abf56e071b03316b5301bbf9
Status: Downloaded newer image for kasproject/kas-isar:2.0
2024-05-10 08:58:49 - INFO     - kas 2.0 started
2024-05-10 08:58:49 - INFO     - /repo$ git rev-parse --show-toplevel
2024-05-10 08:58:49 - INFO     - /repo$ git rev-parse --show-toplevel
2024-05-10 08:58:49 - INFO     - Using /repo as root for repository jailhouse
2024-05-10 08:58:49 - INFO     - /work$ git clone -q https://github.com/ilbers/isar /work/isar
2024-05-10 08:58:56 - INFO     - Repository isar cloned
2024-05-10 08:58:56 - INFO     - /repo$ git rev-parse --show-toplevel
2024-05-10 08:58:56 - INFO     - Using /repo as root for repository jailhouse
2024-05-10 08:58:56 - INFO     - /work/isar$ git status -s
2024-05-10 08:58:56 - INFO     - /work/isar$ git rev-parse --verify HEAD
2024-05-10 08:58:56 - INFO     - b106e0ebcb935c7993cc88c6c49d45928793c9c2
2024-05-10 08:58:56 - INFO     - /work/isar$ git checkout -q 619d6d88ac8c745282fd16773d50a466567615b6
2024-05-10 08:58:56 - INFO     - /repo$ git rev-parse --show-toplevel
2024-05-10 08:58:56 - INFO     - Using /repo as root for repository jailhouse
2024-05-10 08:58:56 - INFO     - /work/isar$ /tmp/tmpryomrhkc /work/build
2024-05-10 08:58:56 - INFO     - /repo$ git rev-parse --show-toplevel
2024-05-10 08:58:56 - INFO     - Using /repo as root for repository jailhouse
2024-05-10 08:58:56 - INFO     - /repo$ git rev-parse --show-toplevel
2024-05-10 08:58:56 - INFO     - Using /repo as root for repository jailhouse
2024-05-10 08:58:56 - INFO     - /work/build$ /work/isar/bitbake/bin/bitbake -k -c build mc:qemu-amd64-jailhouse-demo:demo-image
Parsing recipes: 100% |######################################################################################################################################| Time: 0:00:00
Parsing of 31 .bb files complete (0 cached, 31 parsed). 62 targets, 0 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
Initialising tasks: 100% |###################################################################################################################################| Time: 0:00:00
NOTE: No setscene tasks
NOTE: Executing Tasks
WARNING: mc:qemu-amd64-jailhouse-demo:jailhouse-jailhouse-0.12-r0 do_fetch: Failed to fetch URL git://github.com/siemens/jailhouse;branch=next, attempting MIRRORS if available
ERROR: mc:qemu-amd64-jailhouse-demo:jailhouse-jailhouse-0.12-r0 do_fetch: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; unset _PYTHON_SYSCONFIGDATA_NAME; export GIT_PROXY_COMMAND="oe-git-proxy"; export NO_PROXY="*"; export PATH="/work/isar/scripts:/work/isar/bitbake/bin:/usr/sbin:/usr/bin:/sbin:/bin"; export HOME="/tmp/tmpjicxrpvo"; LANG=C git -c core.fsyncobjectfiles=0 clone --bare --mirror git://github.com/siemens/jailhouse /work/build/downloads/git/github.com.siemens.jailhouse --progress failed with exit code 128, output:
Cloning into bare repository '/work/build/downloads/git/github.com.siemens.jailhouse'...
2024/05/10 09:01:12 socat[404] E connect(5, AF=2 20.205.243.166:9418, 16): Connection timed out
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

ERROR: mc:qemu-amd64-jailhouse-demo:jailhouse-jailhouse-0.12-r0 do_fetch: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:do_fetch(d)
     0003:
File: '/work/isar/meta/classes/base.bbclass', lineno: 174, function: do_fetch
     0170:    try:
     0171:        fetcher = bb.fetch2.Fetch(src_uri, d)
     0172:        fetcher.download()
     0173:    except bb.fetch2.BBFetchException as e:
 *** 0174:        raise bb.build.FuncFailed(e)
     0175:}
     0176:
     0177:addtask fetch before do_build
     0178:
Exception: AttributeError: module 'bb.build' has no attribute 'FuncFailed'

ERROR: Logfile of failure stored in: /work/build/tmp/work/jailhouse-demo-amd64/jailhouse-jailhouse/0.12-r0/temp/log.do_fetch.358
ERROR: Task (mc:qemu-amd64-jailhouse-demo:/repo/recipes-jailhouse/jailhouse/jailhouse_0.12.bb:do_fetch) failed with exit code '1'
NOTE: Tasks Summary: Attempted 61 tasks of which 0 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  mc:qemu-amd64-jailhouse-demo:/repo/recipes-jailhouse/jailhouse/jailhouse_0.12.bb:do_fetch
Summary: There was 1 WARNING message shown.
Summary: There were 2 ERROR messages shown, returning a non-zero exit code.
scilogyhunter@scilogyhunter-Standard-PC-i440FX-PIIX-1996:~/jailhouse-images-v0.12/jailhouse-images$

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54
  • 55
  • 56
  • 57
  • 58
  • 59
  • 60
  • 61
  • 62
  • 63
  • 64
  • 65
  • 66
  • 67
  • 68
  • 69
  • 70
  • 71
  • 72
  • 73
  • 74
  • 75
  • 76
  • 77
  • 78
  • 79
  • 80
  • 81
  • 82
  • 83
  • 84
  • 85
  • 86
  • 87
  • 88
  • 89
  • 90
  • 91
  • 92
  • 93
  • 94
声明：本文内容由网友自发贡献，转载请注明出处：【wpsshop博客】
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号