How to Migrate an Existing RAC database to ASM

Posted at Sunday, May 25, 2008
This post covers how to migrate an existing RAC database to use ASM storage. In my case, I use a 2-node, 10gR2 database on Linux that is currently using OCFS2 filesystems for the database's files. I will migrate the database to ASM using ASMLib on top of iSCSI volumes. This example assumes that you have already prepared the relevant storage devices for recognition by the O/S.

Prepare the Storage

As root on one of the hosts, first identify the device names to format:

# ./iscsi-map.sh
Host / SCSI ID SCSI Device Name iSCSI Target Name
---------------- ----------------------- -----------------
2 /dev/sdb asm2
3 /dev/sde asm1
4 /dev/sdc arch
5 /dev/sdf software
6 /dev/sdd logs
7 /dev/sdg ora2
8 /dev/sdi ora1
9 /dev/sdh crs

Format the devices you plan to use for the ASM migration:

# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.


The number of cylinders for this disk is set to 25024.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-25024, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-25024, default 25024):
Using default value 25024

Command (m for help): p

Disk /dev/sdb: 26.2 GB, 26239565824 bytes
64 heads, 32 sectors/track, 25024 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 25024 25624560 83 Linux

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.


# fdisk /dev/sde
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.


The number of cylinders for this disk is set to 25024.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-25024, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-25024, default 25024):
Using default value 25024

Command (m for help): p

Disk /dev/sde: 26.2 GB, 26239565824 bytes
64 heads, 32 sectors/track, 25024 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Device Boot Start End Blocks Id System
/dev/sde1 1 25024 25624560 83 Linux

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Probe each host to pickup the changes via:

# partprobe
# fdisk -l


Configure ASMLib on each host via:

# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting without typing an
answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Fix permissions of Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: [ OK ]
Loading module "oracleasm": [ OK ]
Mounting ASMlib driver filesystem: [ OK ]
Scanning system for ASM disks: [ OK ]

On one node as root, create the ASM Volumes via:

# /etc/init.d/oracleasm createdisk VOL1 /dev/sdb1
Marking disk "/dev/sdb1" as an ASM disk: [ OK ]


# /etc/init.d/oracleasm createdisk VOL2 /dev/sde1
Marking disk "/dev/sde1" as an ASM disk: [ OK ]

On all other nodes, scan to recognize the changes via:

# /etc/init.d/oracleasm scandisks
Scanning system for ASM disks: [ OK ]


# /etc/init.d/oracleasm listdisks
VOL1
VOL2

Create the ASM Instances

Create directories for the clustered ASM instance on shared storage. In my case, I will put it on the same OCFS2 volumes hosting the voting and cluster registry files for CRS.

# mkdir -p /u02/oracle/crs/asm
# mkdir -p /u02/oracle/crs/asm/bdump
# mkdir -p /u02/oracle/crs/asm/cdump
# mkdir -p /u02/oracle/crs/asm/udump
# chown -R oracle:dba /u02/oracle/crs/asm

Create the initial ASM instance parameter file on the shared storage:

# touch /u02/oracle/crs/asm/init+ASM.ora
# vi /u02/oracle/crs/asm/init+ASM.ora


init+ASM.ora

*.asm_diskgroups='DATA','BACKUPS'
*.asm_diskstring='/dev/oracleasm/disks/*'
*.background_dump_dest='/u02/oracle/crs/asm/bdump'
*.cluster_database=true
*.core_dump_dest='/u02/oracle/crs/asm/cdump'
+ASM2.instance_number=2
+ASM1.instance_number=1
*.instance_type='asm'
*.large_pool_size=20M
*.remote_login_passwordfile='exclusive'
*.user_dump_dest='/u02/oracle/crs/asm/udump'

Link to the parameter file from each host:

# su - oracle
# cd $ORACLE_HOME/dbs
# ln -s /u02/oracle/crs/asm/init+ASM.ora init+ASM1.ora


# su - oracle
# cd $ORACLE_HOME/dbs
# ln -s /u02/oracle/crs/asm/init+ASM.ora init+ASM2.ora

Create the passwordfile on shared storage and link to it from each ASM instance:

# orapwd file=/u02/oracle/crs/asm/orapw+ASM password={password}


Link to the passwordfile on each node:

# cd $ORACLE_HOME/dbs
# ln -s /u02/oracle/crs/asm/orapw+ASM orapw+ASM1


# cd $ORACLE_HOME/dbs
# ln -s /u02/oracle/crs/asm/orapw+ASM orapw+ASM2

Update /etc/oratab on each node:

# echo "+ASM1:/u01/app/oracle/product/10.2:N" >> /etc/oratab


# echo "+ASM2:/u01/app/oracle/product/10.2:N" >> /etc/oratab

Create the ASM Diskgroups:

# export ORACLE_SID=+ASM1
# . oraenv
# sqlplus "/ as sysdba"

SQL*Plus: Release 10.2.0.3.0 - Production on Sun May 25 16:52:56 2008

Copyright (c) 1982, 2006, Oracle. All Rights Reserved.

Connected to an idle instance.

SQL> startup mount;
ASM instance started

Total System Global Area 100663296 bytes
Fixed Size 1260312 bytes
Variable Size 74237160 bytes
ASM Cache 25165824 bytes
ORA-15032: not all alterations performed
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA"
ORA-15063: ASM discovered an insufficient number of disks for diskgroup
"BACKUPS"

SQL> create diskgroup DATA external redundancy disk '/dev/oracleasm/disks/VOL1';

Diskgroup created.

SQL> create diskgroup BACKUPS external redundancy disk '/dev/oracleasm/disks/VOL2';

Diskgroup created.

Now that you have confirmed that the ASM instance is functional, create the spfile from pfile:

SQL> create spfile='/u02/oracle/crs/asm/spfile+ASM.ora' from pfile;

File created.

Update each node to reflect use of the spfile:

# cd $ORACLE_HOME/dbs
# vi init+ASM1.ora

init+ASM1.ora

spfile='/u02/oracle/crs/asm/spfile+ASM.ora'

# cd $ORACLE_HOME/dbs
# vi init+ASM2.ora

init+ASM2.ora

spfile='/u02/oracle/crs/asm/spfile+ASM.ora'

Register the clustered ASM instance with srvctl

# srvctl add asm -n rac1 -i +ASM1 -o $ORACLE_HOME
# srvctl add asm -n rac2 -i +ASM2 -o $ORACLE_HOME
# srvctl enable asm -n rac1 -i +ASM1
# srvctl enable asm -n rac2 -i +ASM2

Start and verify ASM instances:

# srvctl start asm -n rac1
# srvctl start asm -n rac2
# srvctl status asm -n rac2
ASM instance +ASM2 is running on node rac2.

# srvctl status asm -n rac1
ASM instance +ASM1 is running on node rac1.

Prepare to migrate the database in question to use ASM as storage

Query the names of the database files:

SQL> select name from v$datafile;

NAME
--------------------------------------------------------------------------------
/u02/oracle/ora1/em/system01.dbf
/u02/oracle/ora1/em/undotbs01.dbf
/u02/oracle/ora1/em/sysaux01.dbf
/u02/oracle/ora1/em/users01.dbf
/u02/oracle/ora1/em/example01.dbf
/u02/oracle/ora1/em/undotbs02.dbf
/u02/oracle/ora2/em/mgmt.dbf
/u02/oracle/ora1/em/mgmt_ecm_depot1.dbf

8 rows selected.

SQL> select name from v$controlfile;

NAME
--------------------------------------------------------------------------------
/u02/oracle/ora1/em/control01.ctl
/u02/oracle/ora2/em/control02.ctl

SQL> select * from gv$logfile;

INST_ID GROUP# STATUS TYPE MEMBER IS_
---------- ---------- ------- ------- ----------------------------------- ---
1 2 ONLINE /u02/oracle/ora1/em/redo_1_2a.log NO
1 2 ONLINE /u02/oracle/ora2/em/redo_1_2b.log NO
1 1 ONLINE /u02/oracle/ora1/em/redo_1_1a.log NO
1 1 ONLINE /u02/oracle/ora2/em/redo_1_1b.log NO
1 3 ONLINE /u02/oracle/ora1/em/redo_2_3a.log NO
1 3 ONLINE /u02/oracle/ora2/em/redo_2_3b.log NO
1 4 ONLINE /u02/oracle/ora1/em/redo_2_4a.log NO
1 4 ONLINE /u02/oracle/ora2/em/redo_2_4b.log NO
2 2 ONLINE /u02/oracle/ora1/em/redo_1_2a.log NO
2 2 ONLINE /u02/oracle/ora2/em/redo_1_2b.log NO
2 1 ONLINE /u02/oracle/ora1/em/redo_1_1a.log NO

INST_ID GROUP# STATUS TYPE MEMBER IS_
---------- ---------- ------- ------- ----------------------------------- ---
2 1 ONLINE /u02/oracle/ora2/em/redo_1_1b.log NO
2 3 ONLINE /u02/oracle/ora1/em/redo_2_3a.log NO
2 3 ONLINE /u02/oracle/ora2/em/redo_2_3b.log NO
2 4 ONLINE /u02/oracle/ora1/em/redo_2_4a.log NO
2 4 ONLINE /u02/oracle/ora2/em/redo_2_4b.log NO

16 rows selected.

Migrate the Database to ASM

Update the control_files parameter in preparation for migration:

$ srvctl stop database -d em
[oracle@rac1 dbs]$ export ORACLE_SID=em1
[oracle@rac1 dbs]$ . oraenv
[oracle@rac1 dbs]$ sqlplus "/ as sysdba"

SQL*Plus: Release 10.2.0.3.0 - Production on Sun May 25 17:38:38 2008

Copyright (c) 1982, 2006, Oracle. All Rights Reserved.

Connected to an idle instance.

SQL> startup nomount;
ORACLE instance started.

Total System Global Area 268435456 bytes
Fixed Size 1261300 bytes
Variable Size 243269900 bytes
Database Buffers 16777216 bytes
Redo Buffers 7127040 bytes

SQL> alter system set control_files='+DATA/control.ctl' scope=spfile sid='*';

System altered.

SQL> shutdown immediate;
ORA-01507: database not mounted
ORACLE instance shut down.

Migrate the controlfile:

# rman target=/

Recovery Manager: Release 10.2.0.3.0 - Production on Sun May 25 17:40:41 2008

Copyright (c) 1982, 2005, Oracle. All rights reserved.

connected to target database (not started)

RMAN> startup nomount;

Oracle instance started

Total System Global Area 268435456 bytes

Fixed Size 1261300 bytes
Variable Size 247464204 bytes
Database Buffers 12582912 bytes
Redo Buffers 7127040 bytes

RMAN> restore controlfile from '/u02/oracle/ora1/em/control01.ctl';

Starting restore at 25-MAY-08
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=145 instance=em1 devtype=DISK

channel ORA_DISK_1: copied control file copy
output filename=+DATA/control.ctl
Finished restore at 25-MAY-08

Migrate the Datafiles:

RMAN> alter database mount;

database mounted
released channel: ORA_DISK_1

RMAN> backup as copy database format '+DATA';

Starting backup at 25-MAY-08
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=145 instance=em1 devtype=DISK
channel ORA_DISK_1: starting datafile copy
input datafile fno=00007 name=/u02/oracle/ora2/em/mgmt.dbf
output filename=+DATA/em/datafile/mgmt_tablespace.257.655667175 tag=TAG20080525T174614 recid=10 stamp=655668164
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:16:35
channel ORA_DISK_1: starting datafile copy
input datafile fno=00001 name=/u02/oracle/ora1/em/system01.dbf
output filename=+DATA/em/datafile/system.258.655668171 tag=TAG20080525T174614 recid=11 stamp=655668229
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:01:05
channel ORA_DISK_1: starting datafile copy
input datafile fno=00003 name=/u02/oracle/ora1/em/sysaux01.dbf
output filename=+DATA/em/datafile/sysaux.259.655668235 tag=TAG20080525T174614 recid=12 stamp=655668262
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:35
channel ORA_DISK_1: starting datafile copy
input datafile fno=00006 name=/u02/oracle/ora1/em/undotbs02.dbf
output filename=+DATA/em/datafile/undotbs2.260.655668271 tag=TAG20080525T174614 recid=13 stamp=655668293
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:25
channel ORA_DISK_1: starting datafile copy
input datafile fno=00002 name=/u02/oracle/ora1/em/undotbs01.dbf
output filename=+DATA/em/datafile/undotbs1.261.655668297 tag=TAG20080525T174614 recid=14 stamp=655668309
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:15
channel ORA_DISK_1: starting datafile copy
input datafile fno=00005 name=/u02/oracle/ora1/em/example01.dbf
output filename=+DATA/em/datafile/example.262.655668311 tag=TAG20080525T174614 recid=15 stamp=655668322
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:15
channel ORA_DISK_1: starting datafile copy
input datafile fno=00008 name=/u02/oracle/ora1/em/mgmt_ecm_depot1.dbf
output filename=+DATA/em/datafile/mgmt_ecm_depot_ts.263.655668327 tag=TAG20080525T174614 recid=16 stamp=655668337
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:15
channel ORA_DISK_1: starting datafile copy
input datafile fno=00004 name=/u02/oracle/ora1/em/users01.dbf
output filename=+DATA/em/datafile/users.264.655668341 tag=TAG20080525T174614 recid=17 stamp=655668352
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:15
channel ORA_DISK_1: starting datafile copy
copying current control file
output filename=+DATA/em/controlfile/backup.265.655668357 tag=TAG20080525T174614 recid=18 stamp=655668358
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:03
channel ORA_DISK_1: starting full datafile backupset
channel ORA_DISK_1: specifying datafile(s) in backupset
including current SPFILE in backupset
channel ORA_DISK_1: starting piece 1 at 25-MAY-08
channel ORA_DISK_1: finished piece 1 at 25-MAY-08
piece handle=+DATA/em/backupset/2008_05_25/nnsnf0_tag20080525t174614_0.266.655668361 tag=TAG20080525T174614 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:02
Finished backup at 25-MAY-08

RMAN> switch database to copy;

datafile 1 switched to datafile copy "+DATA/em/datafile/system.258.655668171"
datafile 2 switched to datafile copy "+DATA/em/datafile/undotbs1.261.655668297"
datafile 3 switched to datafile copy "+DATA/em/datafile/sysaux.259.655668235"
datafile 4 switched to datafile copy "+DATA/em/datafile/users.264.655668341"
datafile 5 switched to datafile copy "+DATA/em/datafile/example.262.655668311"
datafile 6 switched to datafile copy "+DATA/em/datafile/undotbs2.260.655668271"
datafile 7 switched to datafile copy "+DATA/em/datafile/mgmt_tablespace.257.655667175"
datafile 8 switched to datafile copy "+DATA/em/datafile/mgmt_ecm_depot_ts.263.655668327"

Open the Database by exiting RMAN and then issuing the following via SQL*Plus:

# SQL> alter database open;

Database altered.

Startup the other instances:

# srvctl start instance -d em -i em2

Create a new Temporary Tablespace on the ASM Diskgroup:

SQL> CREATE TEMPORARY TABLESPACE TEMPTBS TEMPFILE '+DATA' SIZE 200M;

Tablespace created.

SQL> ALTER DATABASE DEFAULT TEMPORARY TABLESPACE TEMPTBS;

Database altered.

SQL> DROP TABLESPACE TEMP INCLUDING CONTENTS AND DATAFILES;

Tablespace dropped.

Create new Online Redo Log Groups:

SQL> alter system archive log all;
alter system archive log all
*
ERROR at line 1:
ORA-00271: there are no logs that need archiving

SQL> alter system archive log stop;

System altered.

Create ASM Directories

# export ORACLE_SID=+ASM1
# . oraenv
# sqlplus "/ as sysdba"

SQL*Plus: Release 10.2.0.3.0 - Production on Sun May 25 18:32:31 2008

Copyright (c) 1982, 2006, Oracle. All Rights Reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options

SQL> alter diskgroup DATA add directory '+DATA/em/log1';

Diskgroup altered.

SQL> alter diskgroup DATA add directory '+DATA/em/log2';

Diskgroup altered.

Create New Online Redo Log Groups for each public thread before dropping the old ones

SQL> alter database add logfile thread 1 group 5
2 ('+DATA/em/log1/log_1_5a.log','+DATA/em/log1/log_1_5b.log') size 10M;

Database altered.

SQL> alter database add logfile thread 1 group 6
2 ('+DATA/em/log1/log_1_6a.log','+DATA/em/log1/log_1_6b.log') size 10M;

Database altered.

SQL> alter database add logfile thread 1 group 7
2 ('+DATA/em/log1/log_1_7a.log','+DATA/em/log1/log_1_7b.log') size 10M;

Database altered.

SQL> alter database add logfile thread 2 group 8
2 ('+DATA/em/log2/log_2_8a.log','+DATA/em/log2/log_2_8b.log') size 10M;

Database altered.

SQL> alter database add logfile thread 2 group 9
2 ('+DATA/em/log2/log_2_9a.log','+DATA/em/log2/log_2_9b.log') size 10M;

Database altered.

SQL> alter database add logfile thread 2 group 10
2 ('+DATA/em/log2/log_2_10a.log','+DATA/em/log2/log_2_10b.log') size 10M;

Database altered.

Drop the Old Online Redo Log Groups using the traditional process, for example:

SQL>  alter system switch logfile;


System altered.

SQL> select group#, thread#, status from v$log;

GROUP# THREAD# STATUS
---------- ---------- ----------------
1 1 INACTIVE
5 1 ACTIVE
6 1 ACTIVE
7 1 CURRENT
8 2 CURRENT
9 2 ACTIVE
10 2 ACTIVE

7 rows selected.

SQL> alter database drop logfile group 1;

Database altered.

Optionally, drop the old datafile images:

# rman target=/

Recovery Manager: Release 10.2.0.3.0 - Production on Sun May 25 19:00:45 2008

Copyright (c) 1982, 2005, Oracle. All rights reserved.

connected to target database: EM (DBID=1167076574)

RMAN> delete copy of database;

using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=124 instance=em1 devtype=DISK

List of Datafile Copies
Key File S Completion Time Ckp SCN Ckp Time Name
------- ---- - --------------- ---------- --------------- ----
19 1 A 25-MAY-08 2754400 25-MAY-08 /u02/oracle/ora1/em/system01.dbf
20 2 A 25-MAY-08 2754400 25-MAY-08 /u02/oracle/ora1/em/undotbs01.dbf
21 3 A 25-MAY-08 2754400 25-MAY-08 /u02/oracle/ora1/em/sysaux01.dbf
22 4 A 25-MAY-08 2754400 25-MAY-08 /u02/oracle/ora1/em/users01.dbf
23 5 A 25-MAY-08 2754400 25-MAY-08 /u02/oracle/ora1/em/example01.dbf
24 6 A 25-MAY-08 2754400 25-MAY-08 /u02/oracle/ora1/em/undotbs02.dbf
25 7 A 25-MAY-08 2754400 25-MAY-08 /u02/oracle/ora2/em/mgmt.dbf
26 8 A 25-MAY-08 2754400 25-MAY-08 /u02/oracle/ora1/em/mgmt_ecm_depot1.dbf

Do you really want to delete the above objects (enter YES or NO)? YES
deleted datafile copy
datafile copy filename=/u02/oracle/ora1/em/system01.dbf recid=19 stamp=655668498
deleted datafile copy
datafile copy filename=/u02/oracle/ora1/em/undotbs01.dbf recid=20 stamp=655668498
deleted datafile copy
datafile copy filename=/u02/oracle/ora1/em/sysaux01.dbf recid=21 stamp=655668498
deleted datafile copy
datafile copy filename=/u02/oracle/ora1/em/users01.dbf recid=22 stamp=655668498
deleted datafile copy
datafile copy filename=/u02/oracle/ora1/em/example01.dbf recid=23 stamp=655668498
deleted datafile copy
datafile copy filename=/u02/oracle/ora1/em/undotbs02.dbf recid=24 stamp=655668498
deleted datafile copy
datafile copy filename=/u02/oracle/ora2/em/mgmt.dbf recid=25 stamp=655668498
deleted datafile copy
datafile copy filename=/u02/oracle/ora1/em/mgmt_ecm_depot1.dbf recid=26 stamp=655668498
Deleted 8 objects

Migrate the Archive Log Destinations to ASM:

SQL> alter system set log_archive_dest_1='LOCATION=+BACKUPS' scope=BOTH sid='*';

System altered.

SQL> alter system archive log start;

System altered.

SQL> alter system archive log current;

System altered.

SQL> archive log list
Database log mode Archive Mode
Automatic archival Enabled
Archive destination +BACKUPS
Oldest online log sequence 342
Next log sequence to archive 344
Current log sequence 344

At this point, you should be able to query an ASM instance and find archivelogs in the BACKUPS diskgroup:

SQL> select count(*) from v$asm_file where type='ARCHIVELOG' and group_number = (select group_number from v$asm_diskgroup where name='BACKUPS');

COUNT(*)
----------
4



Labels: , ,