DBM-00201: EDV services STARTUP failed
Sven Illert -
At the beginning of the year I started a new project to setup an Exadata X10M and it’s my first setup to use the Exascale infrastructure which is the successor to Automatic Storage Management (ASM) on Exadata. As always there are ups and downs with such a setup, especially in a highly secured environment where different cluster networks are separated at VLAN level and you have different firewalls you need to get punched holes into. Due to such limitations and vacations I only recently had the chance to setup the standby machine and after some startup issues with the network I was able to run the installation until step number 9 of the OEDA install.sh
script. But then the following errors appeared:
# ./install.sh -s 9 -cf ConfDir/customer.xml
Starting EDV on Compute nodes [node1, node2]
Command /usr/sbin/dbmcli -e ALTER DBSERVER STARTUP SERVICES EDV ran as user root and failed on node node2.customer.com
ERROR:
Starting EDV services...
DBM-00201: EDV services STARTUP failed. Error: DBM-01559: Failure for unknown reasons.
The EDV service belongs to the Exascale infrastructure and provides Linux block volume devices from the Exascale storage servers like ADVM did in the past. So I was investigating further in the local alert log at /var/log/oracle/diag/EXC/exc/node2/trace/alert.log
and it became more obscure.
Errors in file /opt/oracle/dbserver/log/diag/EXC/exc/node2/trace/edv_65357_8.trc (incident=49):
ORA-00600: internal error code, arguments: [EbsEGSLibIOPrimaryJob: initialize_edslib], [1], [], [], [], [], [], [], [], [], [], []
Incident details in: /opt/oracle/dbserver/log/diag/EXC/exc/node2/incident/incdir_49/edv_65357_8_i49.trc
EBS error - ORA-600 internal error
Fri Sep 19 08:35:47 2025 941 msec State dump completed for edv<65357> after ORA-600 occurred
Looking into the trace files didn’t reveal any obvious information so I started to check the configuration and after I some basic network connectivity checks (like from the cluster vm to the storage servers, etc) I had a look at the configuration of the management endpoint of the Exascale RESTful Services (ERS). It took a while to find out what was configured at the virtual machine level. Fortunately the wallet for the secure fabric was already configured and so I was able to use escli
to identify the REST endpoint. This command prints the necessary information already at start, but you can also access configuration details using the lswallet
command.
# escli
Wallet: /etc/oracle/cell/network-config/eswallet/cwallet.sso
User: gridcluster-clu1
ExaCTRL server address: 10.1.1.215:5052
@>lswallet --attributes restEndPoint
restEndPoint
10.1.1.215:5052
I checked with the firewall guys and we learned that this IP adress was initially dedicated for this purpose but was changed later, and the endpoint should be configured using another IP adress. That was the reason why the EDV service didn’t come up. So I had to fix that endpoint address to get the remaining steps done. First the configuration needs to be changed and that can be done using oedacli
.
# cd /EXAVMIMAGES/onecommand/linux-x64
# ./oeadacli
oedacli> load file name=/EXAVMIMAGES/onecommand/linux-x64/ConfDir/customer.xml
oedacli> alter exascalecluster vip=exarestendpoint.customer.com ip=10.1.1.222 where name=clu1
oedacli> save action
oedacli> merge actions
oedacli> deploy actions
oedacli> save file
The deploy actions
command doesn’t really apply the changes to the configuration. But it can be easily applied afterwards by running step 3 of the oeda installation script.
# ./install.sh -s 3 -cf ConfDir/customer.xml
That’s it. It was a bit funny to see an ORA-600
in a non-database context and it wasn’t easy to find a solution to that problem. Hope this might help someone in future, even if it’s the right IP address, but maybe the same error occurs when the port isn’t opened at the firewall level or the endpoint isn’t running somewhere or thing like that.