Oracle Cloud Infrastructure, or OCI, can be a daunting environment for the most experienced technical professional. While it is exceedingly quick to build out an entire infrastructure, and administrative tasks that used to be measure in months can now be done in days, finding our where or why something is not working can still have the twists and turns of a detective novel:
My subject for today is troubleshooting connectivity issues, and especially to an Oracle database. Oracle typically runs on port 1521 for transactional connections. In most on-premise environments, this port was 'open' to traffic on the network specifically and intentionally a long time ago. However, as delivered, OCI locks down all ports unless explicitly open to traffic between each device. Doing this up front used to be the standard some years back, but with a Cloud implementation, you want to be very careful about opening ports to all traffic due to security and audit concerns. This has been flagged as an ongoing concern due to famous incidents of hacking and ransomware exploits - as described on this website.
I recently had two projects where, although this was identified, and the network admins claimed to have opened traffic on this port between all recognized and identified "instances" (usually virtualized servers), we were getting connection errors, such as, from an Oracle SOA Suite installation:
'Unable to connect to the database. Check if DB connection details entered are correct'.
No error codes, no specific details as to what went wrong. We verified everything was correct, and could connect from a client to the database over port 1521 without issue. This seemingly proved the port was open, but......not between this specific port and target. After much deliberation, and assurances from network teams that the ports were open, we did some more detective work.
There are many ways to validate port connectivity, but the tool of choice for us is called netcat. This free tool is available for both linux and Windows, and has a simple syntax. It can also be abbreviated as ncat or just 'nc'.
What is nice about this utility is that it is easily installed (root permission needed in linux, admin permission for Windows), and will be available to run in mere seconds. It might be a wise standard to include it with a hardened OS build, as it only connects if allowed by security audit rules, and reports back if unsuccessful. Here is an installation tutorial.
I will also mention, should it not be accepted as permanent, it is easily uninstalled as well.
Simple is as simple does. Or not!!!! In our use cases, this tool proved to our network administration team that port 1521 was not open system-wide, and they quickly took steps to correct this. It's probably safest to open it only for specified source and targets, due to security concerns.
Once installed, here is the syntax to validate a port as open, and the results:
1-michael.fontana@10.31.2.5's password:
Last failed login: Sat Jul 24 13:05:33 EDT 2021 from 10.31.2.84 on ssh:notty
There was 1 failed login attempt since the last successful login.
Last login: Wed Jul 21 16:38:01 2021 from 10.31.2.84
0(1-michael.fontana@testserver1 [~])$ sudo su -
Last login: Fri Jul 23 15:33:07 EDT 2021 on pts/0
## - You are now root - ##
0(root@testserver1 ~)$ nc -vz testserver5 1521
Connection to 10.31.2.105 1521 port [tcp/ncube-lm] succeeded!
0(root@testserver1 ~)$ nc -vz testserver3 1521
Connection to 10.31.2.103 1521 port [tcp/ncube-lm] succeeded!
0(root@testserver1 ~)$ nc -vz testserver4 1521
Connection to 10.31.2.104 1521 port [tcp/ncube-lm] succeeded!
0(root@testserver1 ~)$ #An example of a failure:
The command will hang if the port is not open. You can use options like -w (wait) to return an error code after a few seconds if you want to use netcat for scripting or other purposes.
