Veritas verify error disk

veritas verify error disk

Disable = Most operations cannot be be used. Booted = Normal startup while using boot disk group. vxrelocd, Monitors for failure events and relocates failed. Error: 'AuthenticateDownload file size check failed: 0= Disk Space checks did not automatically add after installing new. Veritas Data error (cyclic redundancy check). · Contaminated read/write heads of the tape device: · Bad media as there may be hard read/write.

Important: Veritas verify error disk

Veritas verify error disk
Runtime error 5 left 4 dead
SILVERLIGHT 64 C ERROR
Motorola razr v3 critical error 1201
Xerox 3100 printer error mirror

Veritas verify error disk - very

fabric ] Disk Comment vxedit -g <group> set comment="" <disk> Private region problem

## Here i am fixing a "online altused" issue but search the web for different issues as there are a number of ways to the same thing, just remember the vxprivutil command

## obtain the disks private region tag 15
prtvtoc /dev/rdsk/c2t0d0s2

## dd the content sto a file
dd if=/dev/rdsk/c2t0d0s3 of=/tmp/c2t0d0s3_privreg

## run the below script
fix_script

## if now errors reported above and in the file, then dd back to disk
dd if=/tmp/c2t0d0s3_sprers.eu of=/dev/rdsk/c2t0d0s3

## repeat for any other bad disks

Create a disk groupvxdg init group <disk>=<device>
vxdg init <group> <disk>=<device> cds=off    - initialize a non-cds disk group Remove a groupvxdg destroy <group>Add a disk to a groupvxdg –g <group> adddisk <disk>=<device>Remove a disk from a groupvxdg –g <group> rmdisk <disk>Replace failed disk

vxdg -k -g <group> adddisk <disk>=<device>

-k = forces vxvm to take media name of the failed disk & assign it to the new disk

Import a group

vxdg import <group>
vxdg -n <new-group-name> import <old-group-name>
vxdg -C import <group>

Import group (clear any flags)

vxdg import -C <group>

-C - clears any exist host flags

Deport a groupvxdg deport <group>
vxdg -n <new-group-name> deport <old_group_name> List no hot use on diskvxdg –g <group> nohotuseList spare space on diskvxdg –g <group> spareDisplay free spacevxdg –g <group> freeBackup disk group (vxvm ) vxconfigbackupRestore disk group (vxvm ) vxconfigrestore [-p mirror-stripe]

stripe-mirror = layered volume
concat-mirror = layered volume
mirror-concat = non-layered volume
mirror-stripe = non-layered volume

Mirror a simple volume vxassist mirror <volume> <disk>
vxassist -g <group> remove mirror <vol> !disk01    - remove the disk01 mirror Create a stripped volume

vxassist make <volume> <size> layout=stripe

vxassist -g <group> -o ordered make <vol> <size> layout=stripe ncol=3 <disk1> <disk2> <disk3>

Create mirrored volume with log vxassist make <volume> <size> layout=mirror, log nmirror=# nlog=#

Note: for information about logging see logging section below Create a raid volume vxassist make <volume> <size> layout=raid5 Create a raw volume

vxassist -U <usage_type> make <volume> <size> alloc='<disk>'

types:
fsgen - filesystems
gen - raw volumes
raid - supports raid5
root - suuports root filesystems for booting
swap - performs no recovery on startup
relayout - used temporary for disk relayout operations

Remove a volume

vxedit –rf rm <volume>
vxassist -g <group> remove volume <vol>

Note: you must disable the volume first

Initializing a volume vxvol init state <volume> [plex]state=clean,enable,active Online Relayout

vxassist -g <group> relayout <vol> layout=stripe ncol=2
vxassist -g <group> relayout <vol> layout=stripe ncol=+1
vxassist -g <group> relayout <vol> layout=stripe ncol=-1
vxassist -g <group> relayout <vol> layout=stripe stripe=32k ncol=5
vxassist -g <group> relayout <vol> layout=raid5 stripeunit=32k ncol3
vxassist -g <group> convert <vol> layout=stripe-mirror

# Display the relayout operation
vxrelayout -g <group> [status od -v -t c

Note: For a hexadecimal dump, replace "c" with "x."


Example, with typical output:


 AIX
Syntax:

dd if=/dev/<os_native_name> count=1 bs=

Troubleshooting Failing Disks, Missing Disks and the "failed was" status

Problem


This article contains a procedure for troubleshooting VxVM managed disks reporting a "failed" or "failed was" state, as displayed by the VxVM vxdisk command.

 

Error Message

 

In the below example, VxVM disk named "disk4" is reported as "failed was":
 


Solution

 

Table of Contents

1. Introduction
2. "Failed" versus "failing" disks
3. Making an emergency backup of the disk group configuration
4. Has the disk been excluded in sprers.eue?
5. Has the disk been overwritten by another logical volume manager solution?
6. Determining if a disk can be reattached
7. Verifying that a disk is readable to the operating system
8. Have the paths to the disk been disabled?
9. Restoring the disk group configuration using vxconfigrestore
Restoring the disk group configuration manually, using UDIDs and Disk IDs
Restarting the volume

 

1. Introduction

(Back to top)
 

VxVM managed disks may report a "failed" or "failed was" state, as displayed by the vxdisk command.


A "failed" status is a record of a disk that is no longer accessible.

This is often caused by sustained I/O errors to the disk that prevents it from being read by the operating system (OS). It may also be the result of corruption within the Veritas private region.

The private region is the portion of the disk where Veritas stores records about the disk group, such as disks, volumes, subdisks and plexes. This can be contrasted with the public region, which contains the actual volumes, including user data.




 

2. "Failed" versus "failing" disks

(Back to top)


The status of "failed" should not be confused with a status of "failing" This article primarily discusses the "failed" status, as reported by vxdisk.

For information on troubleshooting the "failing" status, see sprers.eu




3. Making an emergency backup of the disk group configuration

(Back to top)

Before making any changes to the disks or the disk group configuration, create a backup of the disk group configuration using the vxconfigbackup command.

Vxconfigbackup does not back up any actual data that is contained within the volumes. Instead, it backs up the Veritas private region configuration database that resides on the disks associated with the diskgroup, along with some information about the disks themselves. The configuration database stores information about which disks are contained by the disk group, volume structures, plexes and subdisks.

If vxconfigbackup is not available, vxprivutil can be used to dump a copy of the configuration database.
 


More details about vxconfigbackup and vxprivutil, including syntax and examples, can be found in this article:

"Using vxconfigbackup and vxprivutil to back up the disk group configuration of the Veritas private region"
sprers.eu


 

 

4. Has the disk been excluded by sprers.eue?

(Back to top)


The /etc/vx/sprers.eue file maintains a list of paths, controllers and products that are excluded from Veritas Volume Manager's (VxVM's) view.

Check if the disk, or its associated paths or controllers are listed in this file.

If the value of "exclude_all" is 1, all devices will be excluded.


Figure 1 - Default contents of /etc/vx/sprers.eue

 


exclude_all 0

paths

#

controllers

#

product

#
 

 

5. Has the disk been overwritten by another logical volume manager solution?

(Back to top)


If vxdisk shows that the disk type includes the words "LVM" or "ZFS," then the disk may have been overwritten by another logical volume manager (LVM) solution. It is also possible that there is a problem with the SAN zoning which may have caused disks to be presented to the wrong systems. Before making any further changes, ensure that the disk is not supposed to be zoned to another system.

Please be aware that if a disk has been overwritten by another logical volume manager, there is a chance that the public region of the disk might have been affected. If the public region was modified, then it is usually impossible to restore the data for that filesystem.

Once the disk is removed from the LVM solution, it will need to be initialized for Veritas use, using the vxdisksetup command. Care should be taken when initializing the disk as default private region sizes and public region starting offsets can vary per VxVM version. Please consult Veritas Technical Support for guidance. Refer to the documentation for the appropriate vendor for information about removing a disk from the control of their LVM solution.





6. Determining if a disk can be reattached

(Back to top)

The vxreattach command is used to reattach the original disk media name with the specified disk group. It can normally only be used if the status of the VxVM disk is shown in an "online" state by vxdisk list (see Figure 2).

Run vxreattach, using the "-c" argument, to determine if a disk can be reattached to the disk group.


Figure 2 - Using vxreattach, with the "-c" argument, to check if a reattach is possible


Syntax:

vxreattach -c <disk_media_name>


Example, with typical output:

 


In this case, "datadg" is the name of the disk group while "datadg02" is the disk media name, as shown by vxdisk.


 




If vxreattach -c returns a disk group and disk media name, without returning any errors, proceed with reattaching the disk (Figure 3). If a reattach is not possible, a V error will appear.


Figure 3 - Using vxreattach to reattach a disk to the disk group


Syntax:

vxreattach -br <disk_media_name>


Example, with typical output:


Notice that vxdisk now shows a disk media name, "datadg02," for disk_4.


 

7. Verifying that a disk is readable to the operating system

(Back to top)


Use native OS commands to confirm that the OS can read the disk, including the disk label.

 

  • Use commands, such as prtvtoc, fdisk, lspv or diskinfo to read the disk label.
  • Use dd to read a block from the disk.

Veritas depends on the OS device drivers to communicate with disks. If the OS is unable to read a disk, Veritas will also fail to read it. If a disk does not have a label, or the label has been corrupted, Veritas will not recognize the disk. Completing these steps will assist with identifying the source of a disk outage.
 


More details, including syntax and examples, can be found in this article:

"Verifying that a disk is readable by the OS"
sprers.eu


8. Have the paths to the disk been disabled?

(Back to top)

 

Use vxdmpadm to determine the status of the paths to the disks (Figure 4).

Veritas will disable a path if serious or sustained I/O errors occur. When all paths to a disk are disabled, the server will be unable to read or write to the volume. If a path has been disabled, review the syslog for events that are reported by "vxdmp," or "scsi" for I/O errors.

Although a path can be re-enabled using "vxdmpadm enable," vxdmp should automatically evaluate the status of a path in five minute intervals using a scsi inquiry. If the query is successful, the path is automatically re-enabled. If a path remains disabled beyond this interval, it is possible that I/O errors are still being detected, warranting further investigation. Paths are not automatically re-enabled If the diskgroup has been disabled, or if vxesd is stopped. The behavior of vxdmp in response to disabled paths can be modified via the DMP tunables, which can be viewed using "vxmpadm gettune."
 


Note: Although the syslog may show that vxdmp is the source of an I/O error, vxdmp itself is not usually the origin. Veritas depends on the OS device drivers to communicate with disks. When I/O errors occur, they are reported to Veritas by the device drivers. Vxdmp will report the errors that have been passed to it by the device drivers and may disable a path in response to the events.




Figure 4 - Example of a disabled path, as reported by vxdmpadm


Syntax:

vxdmpadm getsubpaths


For example:




 


To enable a disabled path, run:

# vxdmpadm enable path=<device>

In this instance, sdi and sdl are shown as disabled

# vxdmpadm enable path=sdi,sdl

 

9. Restoring the disk group configuration using vxconfigrestore

(Back to top)

If a vxreattach is not possible, you need to use vxconfigrestore to recover the disk group configuration, consult with Veritas Technical Support before doing so.

vxconfigrestore does not restore the actual data that is contained within the volumes. It only restores the Veritas disk group configuration database that is located within the private region of the disks. The configuration database stores information about which disks are contained by the disk group, volume structures, plexes and subdisks.
 


More details about vxconfigrestore, including syntax and examples, can be found in this article:

"Restoring the disk group configuration using vxconfigrestore"
sprers.eu


 

Restoring the disk group configuration manually, using UDIDs and Disk IDs

(Back to top)

If using vxconfigrestore is not possible, another method for recovering the disks is to compare the UDID or Disk ID attributes of the disks with the records that are contained with the private region configuration database.
 


More details about comparing UDIDs and Disk IDs, including syntax and examples, can be found in this article:

"Restoring the disk group configuration manually, using udids or disk IDs"
sprers.eu


Restarting the volume

(Back to top)


Once the original failed disk has been added back to its disk group, additional steps may be needed to recover the volume. Use the vxprint command to determine the current status of all volumes in a disk group (Figure 5).

  • For mirrored volumes:
    • If at least one plex was not affected by the outage, the other plexes should be resynchronized when they are reattached to the volume.  It may be necessary to use vxrecover to initiate this process (Figure 6).
    • If all plexes were affected by the outage, it may be necessary to manually review each plex to determine which contain the most recent updates.

Warning: Do not simply force-start a mirrored volume. This may cause a plex that contains old, or corrupt, blocks to overwrite a plex that contains up-to-date data. A procedure for manually determining the most up-to-date mirror plex can be found in this article:

"Manually determining which mirror plex contains the most recent data and then resynchronizing"
sprers.eu


  • For non-mirrored volumes:
    • It may be necessary to manually force restart the volume using vxvol after adding the disk back to the disk group (Figure 5).



Figure 5 - Using vxprint to determine the status of a volume


Syntax:

vxprint -g <disk_group> -ht


Example, with typical output:

In this case, vxprint shows that the volume "vol1" is disabled. The plex status is "IOFAIL," which indicates that a sustained I/O interruption to the volume has occurred. After the associate disk is added back to the disk group, the volume will need to be restarted manually using vxvol.





 




Figure 6 - Using vxrecover to start volume recovery and resynchronization of a volume

The following vxrecover syntax will attempt to recover all volumes in a disk group


Syntax:

# vxrecover  -g <disk-group>  -sb 

It is also possible to recovery a single volume



Prior to running the vxrecover command, the current vxprint shows plex "vol" as "DISABLED" for volume "vol1".






The vxprint output now shows both plexes for volume "vol1" as "ACTIVE."

 

 


Figure 7 - For volumes not started by the vxrecover operation, use the vxvol command instead. Volumes can be be manually started using vxvol to start a volume and using vxprint to review any changes in the status of the volume
 


In this instance, VxVM volume "vol1" only has a single plex (not mirrored), so will need to be force started. Do not force start mirrored volumes.

Syntax:

# vxvol -fg <disk_group> <volume-name>

# vxvol -fg datadg start vol1

Sample output:

The volume "vol1" is now shown by vxprint as started (ENABLED).



 


Is it possible to start all volumes in a disk group by typing:

 

 

keywords: failed, failing, failed disk, failed disks, failing disk, failed disks,

grep -i version Upgrade disk version vxdg upgrade <group>                       - upgrade to current version
vxdg -T 50 upgrade <group>                 - upgrade to version 50
vxdg -T 50 init <group> <disk>=<device>    - creater new group @ version 50 Boot/Default DG vxdg bootdg
vxdg defaultdg

vxdctl defaultdg <group> - set defaultdg
Adding mirror to root /etc/vx/bin/vxrootmir <alternate> create rootvol, swap vol, Create a simple volume

vxassist make <volume> <size> <disk>
vxassist -g <group> make <vol> <size> !ctrl:c2     - don't use controller 2

vxassist make <volume> <size> <disk> layout=[stripe-mirror more

Logging help in recovery and can speed it up dramatically, the main form of logging in veritas is the DRL (dirty region log) which performs the following

Policies can be used if you have slower disks within a volume and you wish to use the faster disks.

Veritas have created some scripots that can check the integrity of the vxvm setup i.e mirrored volumes, spares, etc. The scripts are based on rules and there are a number of differents rules veritas has set, look in the rules directory to see all of them.

Display Description vxse_raid5log1 info
Check rules vxse_raid5log1 -g <group> check
List spare rules vxse_spares list
Run spare run

vxse_spares run

Note: you need to run "/etc/init.d/isisd start" to start the necessary daemons first

VXSE Paths

Rules/opt/VRTS/vxse/vxvm
Default Parameters /etc/default/vxse

VxDMP

see VxDMP for more information

Veritas links