oracle磁盘组实例启动,12c rac 实例无法启动之磁盘组空间耗尽

一、环境描述

oracle 12c rac

二、处理过程

1、实例启动失败

SYS@orcl1> startup

ORACLE instance started.

Total System Global Area 1241513984 bytes

Fixed Size 2923872 bytes

Variable Size 620757664 bytes

Database Buffers 603979776 bytes

Redo Buffers 13852672 bytes

Database mounted.

ORA-03113: end-of-file on communication channel

Process ID: 8482

Session ID: 1 Serial number: 48907

2、查看日志

部分省略...............

Starting background process ARC2

Starting background process ARC3

ARC2 started with pid=44, OS id=8737

Unable to create archive log file '+DATADG'

Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_8659.trc:

ORA-19504: failed to create file "+DATADG"

ORA-17502: ksfdcre:4 Failed to create file +DATADG

ORA-15041: diskgroup "DATADG" space exhausted

ARCH: Error 19504 Creating archive log file to '+DATADG'

Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_8659.trc:

ORA-16038: log 2 sequence# 108 cannot be archived

ORA-19504: failed to create file ""

ORA-00312: online log 2 thread 1: '+DATADG/ORCL/ONLINELOG/group_2.263.954539539'

ARC3 started with pid=45, OS id=8739

System state dump requested by (instance=1, osid=8659), summary=[abnormal instance termination].

System State dumped to trace file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_diag_8552_20171231165944.trc

USER (ospid: 8659): terminating the instance due to error 16038

Dumping diagnostic data in directory=[cdmp_20171231165944], requested by (instance=1, osid=8659), summary=[abnormal insta

nce termination].

Instance terminated by USER, pid = 8659

3、确认磁盘组空间大小

$asmcmd -p

ASMCMD [+] > lsdg

State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name

MOUNTED EXTERN N 512 4096 1048576 10240 5763 0 5763 0 Y CRSDG/

MOUNTED EXTERN N 512 4096 1048576 6144 13 0 13 0 N DATADG/

ASMCMD [+] >

4、处理思路

1、使用rm命令清理部分归档日志。

2、启动数据库

3、使用rman crosscheck archivelog all;

4、扩磁盘组

5、注意

需要使用rman进行crosscheck ,然后再删除归档日志在控制文件中的记录。

RMAN> crosscheck archivelog all;

using target database control file instead of recovery catalog

allocated channel: ORA_DISK_1

channel ORA_DISK_1: SID=66 instance=orcl2 device type=DISK

validation failed for archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_50.280.962120097 RECID=11 STAMP=962120097

validation failed for archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_51.282.962121355 RECID=13 STAMP=962074554

validation failed for archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_52.283.962121355 RECID=14 STAMP=962074555

validation failed for archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_53.288.962121385 RECID=19 STAMP=962121385

validation failed for archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_54.289.962184681 RECID=20 STAMP=962184682

validation failed for archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_55.291.962185743 RECID=22 STAMP=962185742

validation failed for archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_56.292.962188105 RECID=23 STAMP=962188106

validation failed for archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_57.295.962188239 RECID=26 STAMP=962188238

validation failed for archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_58.297.962188923 RECID=28 STAMP=962188924

部分省略..................................

validation succeeded for archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_31/thread_2_seq_109.288.964199163 RECID=132 STAMP=964199165

validation succeeded for archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_31/thread_2_seq_110.325.964199595 RECID=135 STAMP=964199601

Crosschecked 124 objects

RMAN>

RMAN> delete expired archivelog all;

released channel: ORA_DISK_1

allocated channel: ORA_DISK_1

channel ORA_DISK_1: SID=66 instance=orcl2 device type=DISK

List of Archived Log Copies for database with db_unique_name ORCL

=====================================================================

Key Thrd Seq S Low Time

------- ---- ------- - --------------------

11 1 50 X 07-DEC-2017 15:24:55

Name: +DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_50.280.962120097

13 1 51 X 07-DEC-2017 15:34:57

Name: +DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_51.282.962121355

14 1 52 X 07-DEC-2017 02:55:53

Name: +DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_52.283.962121355

19 1 53 X 07-DEC-2017 15:56:20

部分省略 ...........................................

51 2 66 X 08-DEC-2017 07:41:03

Name: +DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_2_seq_66.319.962225883

53 2 67 X 08-DEC-2017 20:57:57

Name: +DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_2_seq_67.321.962225885

57 2 68 X 09-DEC-2017 10:16:05

Name: +DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_2_seq_68.325.962273785

59 2 69 X 09-DEC-2017 10:16:23

Name: +DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_2_seq_69.327.962274133

Do you really want to delete the above objects (enter YES or NO)? yes

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_50.280.962120097 RECID=11 STAMP=962120097

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_51.282.962121355 RECID=13 STAMP=962074554

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_52.283.962121355 RECID=14 STAMP=962074555

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_53.288.962121385 RECID=19 STAMP=962121385

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_54.289.962184681 RECID=20 STAMP=962184682

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_55.291.962185743 RECID=22 STAMP=962185742

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_56.292.962188105 RECID=23 STAMP=962188106

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_57.295.962188239 RECID=26 STAMP=962188238

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_58.297.962188923 RECID=28 STAMP=962188924

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_59.298.962206427 RECID=29 STAMP=962206427

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_60.300.962208543 RECID=31 STAMP=962208544

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_61.302.962223761 RECID=48 STAMP=962177843

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_62.305.962223817 RECID=36 STAMP=962223817

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_63.306.962223899 RECID=37 STAMP=962223899

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_64.309.962224067 RECID=40 STAMP=962224066

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_65.310.962224095 RECID=41 STAMP=962224094

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_66.311.962224101 RECID=42 STAMP=962224100

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_67.314.962224111 RECID=45 STAMP=962224111

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_68.316.962224463 RECID=47 STAMP=962224462

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_69.317.962224863 RECID=49 STAMP=962224862

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_70.320.962225885 RECID=52 STAMP=962225884

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_1_seq_71.322.962273773 RECID=54 STAMP=962273772

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_1_seq_72.323.962273775 RECID=55 STAMP=962273775

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_1_seq_73.324.962273779 RECID=56 STAMP=962273783

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_1_seq_74.326.962273785 RECID=58 STAMP=962273785

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_2_seq_47.281.962120099 RECID=12 STAMP=962073300

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_2_seq_48.284.962121379 RECID=15 STAMP=962121380

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_2_seq_49.285.962121381 RECID=16 STAMP=962121380

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_2_seq_50.286.962121383 RECID=17 STAMP=962121382

部分省略 ....................

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_2_seq_66.319.962225883 RECID=51 STAMP=962225883

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_2_seq_67.321.962225885 RECID=53 STAMP=962225884

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_2_seq_68.325.962273785 RECID=57 STAMP=962273784

deleted archived log

archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_2_seq_69.327.962274133 RECID=59 STAMP=962274134

Deleted 48 EXPIRED objects

最近也是有客户遇到类似的问题，通过操作系统命令删除了部分归档，然后备份脚本备份失败，发现是因为这个原因引起的，优化一下脚本，添加进去crosscheck 命令，以后就不会出现类似问题了。