Restoring Vote disk due to ASM disk failures3.docx

1、Restoring Vote disk due to ASM disk failures 3Restoring Vote disk due to ASM disk failures - 3Previous blogs showed the restoring of vote disk due tooneandtwoASM disk failures in the diskgroup containing vote disks. This scenario is when all the disks in the ASM diskgroup containg vote disks are aff

2、ected.Scenario 3.1. Only Vote disks are in ASM diskgroup2. ASM diskgroup has normal redundancy with only three failure groups3. All failure groups are affected4. OCR is located in a separate location (in another diskgroup or block device location - not supported by Oracle only valid during migration

3、 could be moved to after installation for testing purposes)1.Move the vote disks to ASM diskgroupif not done already. Below is the current configurationcrsctl query css votedisk# STATE File Universal Id File Name Disk group- - - - -1. ONLINE 9da0c9d64b9a4fcabf3d143627e7f3e8 (ORCL:CLUS1) CLUSTERDG2.

4、ONLINE 380dba2e5b654f80bfe69d64c5d0c12c (ORCL:CLUS2) CLUSTERDG3. ONLINE 95968700d82b4ff8bf88167a28b080a4 (ORCL:CLUS3) CLUSTERDG2. Identify the disks belonging to the diskgroup and corrupt them to simulate disk failure. One disk being corrupted shown here# /etc/init.d/oracleasm querydisk -p clus3Disk

5、 CLUS3 is a valid ASM disk/dev/sdc10: LABEL=CLUS3 TYPE=oracleasm# dd if=/dev/zero of=/dev/sdc10 count=204800 bs=8192204800+0 records in204800+0 records out3. ocssd.log will show the detection of vote disk corruption2010-08-25 10:33:33.572: CSSD1235470656clssnmvDiskKillCheck: voting disk corrupted (0

6、x00000000,0x00000000) (ORCL:CLUS1)2010-08-25 10:33:33.572: CSSD1235470656clssnmvDiskAvailabilityChange: voting file ORCL:CLUS1 now offline2010-08-25 10:33:34.263: CLSF1193511232Closing handle:0x68d61102010-08-25 10:33:34.263: SKGFD1193511232Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: closin

7、g handle 0x68ae880 fordisk :ORCL:CLUS1:2010-08-25 10:33:34.263: CSSD1193511232clssnmvScanCompletions: completed 4 items2010-08-25 10:33:34.270: CSSD1141061952clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:CLUS3 sched delay 1680 margin 1500 cur_ms 1026610624 lastalive 10266089442010-08-

8、25 10:33:34.270: CSSD1141061952clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:CLUS2 sched delay 1560 margin 1500 cur_ms 1026610624 lastalive 10266090642010-08-25 10:33:34.270: CSSD1141061952clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:CLUS1 sched delay 1720 margin 1500

9、cur_ms 1026610624 lastalive 10266089042010-08-25 10:33:34.572: CLSF1235470656Closing handle:0x2aaab42465602010-08-25 10:33:34.572: SKGFD1235470656Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: closing handle 0x2aaab423b080for disk :ORCL:CLUS1:2010-08-25 10:33:35.042: CLSF1214490944Closing hand

10、le:0x2aaab43557f02010-08-25 10:33:35.042: SKGFD1214490944Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: closing handle 0x2aaab43fd730for disk :ORCL:CLUS1:2010-08-25 10:33:36.268: CSSD1298409792clssnmSendingThread: sending status msg to all nodes2010-08-25 10:33:36.268: CSSD1298409792clssnmSend

11、ingThread: sent 5 status msgs to all nodes2010-08-25 10:33:36.272: CSSD1141061952clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:CLUS1 sched delay 3720 margin 1500 cur_ms 1026612624 lastalive 10266089042010-08-25 10:33:37.044: CSSD1214490944clssnmvDiskOpen: Opening ORCL:CLUS12010-08-25

12、10:33:37.044: SKGFD1214490944Handle 0x2aaab423b080 from lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: for disk :ORCL:CLUS1:2010-08-25 10:33:37.044: CLSF1214490944Opened hdl:0x2aaab41e9170 for dev:ORCL:CLUS1:2010-08-25 10:33:37.054: CSSD1214490944clssnmvStatusBlkInit: myinfo nodename hpc1, uni

13、queness 12826579362010-08-25 10:33:37.054: CSSD1214490944clssnmvDiskAvailabilityChange: voting file ORCL:CLUS1 now online2010-08-25 10:33:37.054: SKGFD1193511232Handle 0x682fcc0 from lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: for disk :ORCL:CLUS1:2010-08-25 10:33:37.054: CLSF1193511232Open

14、ed hdl:0x6906de0 for dev:ORCL:CLUS1:2010-08-25 10:33:37.575: SKGFD1235470656Handle 0x2aaab43fd730 from lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: for disk :ORCL:CLUS1:2010-08-25 10:33:37.575: CLSF1235470656Opened hdl:0x2aaab41ee180 for dev:ORCL:CLUS1:2010-08-25 10:33:37.598: CSSD1235470656

15、clssnmvDiskKillCheck: voting disk corrupted (0x00000000,0x00000000) (ORCL:CLUS1)2010-08-25 10:33:37.598: CSSD1235470656clssnmvDiskAvailabilityChange: voting file ORCL:CLUS1 now offline2010-08-25 10:33:38.055: CLSF1193511232Closing handle:0x6906de02010-08-25 10:33:38.055: SKGFD1193511232Lib :ASM:/opt

16、/oracle/extapi/64/asm/orcl/1/libasm.so: closing handle 0x682fcc0 fordisk :ORCL:CLUS1:2010-08-25 10:33:38.279: CSSD1141061952clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:CLUS1 sched delay 5730 margin 1500 cur_ms 1026614634 lastalive 10266089042010-08-25 10:33:38.601: CLSF1235470656Clo

17、sing handle:0x2aaab41ee1802010-08-25 10:33:38.601: SKGFD1235470656Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: closing handle 0x2aaab43fd730for disk :ORCL:CLUS1:2010-08-25 10:33:38.658: CSSD1214490944clssnmvDiskAvailabilityChange: voting file ORCL:CLUS1 now online2010-08-25 10:33:38.659: SKG

18、FD1193511232Handle 0x682fcc0 from lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: for disk :ORCL:CLUS1:2010-08-25 10:33:38.659: CLSF1193511232Opened hdl:0x68b98f0 for dev:ORCL:CLUS1:2010-08-25 10:33:39.600: SKGFD1235470656Handle 0x2aaab43fd730 from lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/liba

19、sm.so: for disk :ORCL:CLUS1:2010-08-25 10:33:39.600: CLSF1235470656Opened hdl:0x2aaab4276240 for dev:ORCL:CLUS1:2010-08-25 10:33:39.689: CSSD1172531520clssnmvDiskKillCheck: voting disk corrupted (0x00000000,0x00000000) (ORCL:CLUS2)2010-08-25 10:33:39.689: CSSD1172531520clssnmvDiskAvailabilityChange:

20、 voting file ORCL:CLUS2 now offline2010-08-25 10:33:39.830: CLSF1162041664Closing handle:0x68a47302010-08-25 10:33:39.830: SKGFD1162041664Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: closing handle 0x68a4790 fordisk :ORCL:CLUS2:2010-08-25 10:33:40.271: CLSF1183021376Closing handle:0x2aaab43f

21、dd202010-08-25 10:33:40.271: SKGFD1183021376Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: closing handle 0x2aaab43fce20for disk :ORCL:CLUS2:2010-08-25 10:33:40.690: CLSF1172531520Closing handle:0x689e5d02010-08-25 10:33:40.690: SKGFD1172531520Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.s

22、o: closing handle 0x6c43870 fordisk :ORCL:CLUS2:2010-08-25 10:33:41.278: CSSD1298409792clssnmSendingThread: sending status msg to all nodes2010-08-25 10:33:41.278: CSSD1298409792clssnmSendingThread: sent 5 status msgs to all nodes2010-08-25 10:33:41.281: CSSD1141061952clssnmvSchedDiskThreads: DiskPi

23、ngThread for voting file ORCL:CLUS2 sched delay 2450 margin 1500 cur_ms 1026617634 lastalive 10266151842010-08-25 10:33:43.289: CSSD1141061952clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:CLUS2 sched delay 4460 margin 1500 cur_ms 1026619644 lastalive 10266151842010-08-25 10:33:43.832:

24、 CSSD1162041664clssnmvDiskOpen: Opening ORCL:CLUS22010-08-25 10:33:43.832: SKGFD1162041664Handle 0x6c43870 from lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: for disk :ORCL:CLUS2:2010-08-25 10:33:43.832: CLSF1162041664Opened hdl:0x68f9bc0 for dev:ORCL:CLUS2:2010-08-25 10:33:43.876: CSSD116204

25、1664clssnmvStatusBlkInit: myinfo nodename hpc1, uniqueness 12826579362010-08-25 10:33:43.876: CSSD1162041664clssnmvDiskAvailabilityChange: voting file ORCL:CLUS2 now online2010-08-25 10:33:43.876: SKGFD1183021376Handle 0x2aaab43fce20 from lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: for disk

26、 :ORCL:CLUS2:2010-08-25 10:33:43.876: CLSF1183021376Opened hdl:0x2aaab41c1fc0 for dev:ORCL:CLUS2:2010-08-25 10:33:44.698: SKGFD1172531520Handle 0x68a4790 from lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: for disk :ORCL:CLUS2:2010-08-25 10:33:44.698: CLSF1172531520Opened hdl:0x6c57440 for dev

27、:ORCL:CLUS2:2010-08-25 10:33:44.707: CSSD1172531520clssnmvDiskKillCheck: voting disk corrupted (0x00000000,0x00000000) (ORCL:CLUS2)2010-08-25 10:33:44.707: CSSD1172531520clssnmvDiskAvailabilityChange: voting file ORCL:CLUS2 now offline2010-08-25 10:33:44.820: CSSD1162041664clssnmvDiskAvailabilityCha

28、nge: voting file ORCL:CLUS2 now online2010-08-25 10:33:45.289: CSSD1141061952clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:CLUS1 sched delay 1750 margin 1500 cur_ms 1026621644 lastalive 10266198942010-08-25 10:33:50.296: CSSD1298409792clssnmSendingThread: sent 4 status msgs to all nod

29、es2010-08-25 10:33:53.311: CSSD1141061952clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:CLUS1 sched delay 1660 margin 1500 cur_ms 1026629664 lastalive 10266280042010-08-25 10:34:13.342: CSSD1298409792clssnmSendingThread: sent 5 status msgs to all nodes2010-08-25 10:34:14.984: CSSD11410

30、61952clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:CLUS2 sched delay 1560 margin 1500 cur_ms 1026651334 lastalive 10266497742010-08-25 10:34:14.984: CSSD1141061952clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:CLUS1 sched delay 1580 margin 1500 cur_ms 1026651334 lastaliv

31、e 10266497542010-08-25 10:34:16.398: CSSD1224980800clssnmvDiskKillCheck: voting disk corrupted (0x00000000,0x00000000) (ORCL:CLUS3)2010-08-25 10:34:16.398: CSSD1224980800clssnmvDiskAvailabilityChange: voting file ORCL:CLUS3 now offline2010-08-25 10:34:16.830: CLSF1204001088Closing handle:0x68a52202010-08-25 10:34:16.830: SKGFD1204001