Posts

Recovering Crashed VBox VM

My Linux host os disk partition filled up due to .xsession_errors consuming all available space and subsequently the running VBox VMs did not shut down cleanly and would not restart. Below is a link indicating possible fixes for .xsession_errors issue. http://askubuntu.com/questions/177058/xsession-errors-file-is-huge-how-can-i-disable Both the active and previous versions of the vmName.vbox XML files became corrupted and the StorageControllers element contents were missing. I tried adding them from an old backup, but that didn't work since I had moved and rebuilt the VM. Should have had a more recent backup. I was able to recover by remapping the storage in the VirtualBox Manager, but it took several tries until I discovered the correct storage type and attributes. If I had recorded those somewhere, it would have been easier to recover.

ActiveMQ remote JMX

With ActiveMQ 5.3.2 at least, you can't connect remotely to JMX unless the hostname in /etc/hosts maps to the routable IP and not localhost 127.0.0.1 http://kyrill007.livejournal.com/3517.html

ActiveMQ Java 7 JAVA_HOME

Executing the stock ActiveMQ 5.3.2 startup script with java 7 complains about JAVA_HOME Turns out the error was related to .activemqrc that was created when I tested with ActiveMQ 5.8.0. The JAVACMD is set to "auto" in that file part way through the 5.3.2 activemq startup script. apache-activemq-5.3.2/bin$ activemq Error: JAVA_HOME is not defined correctly.   We cannot execute auto Java home was set correctly: ...java/apache-activemq-5.3.2/bin$ echo $JAVA_HOME /home/myusername/java/jdk1.7.0_15 Needed to edit the script and add the following right before "if [ -z "$JAVACMD" ] ; then" JAVACMD= This clears the mysterious setting to auto, echoing it returned a blank line: ...java/apache-activemq-5.3.2/bin$ echo $JAVACMD ...java/apache-activemq-5.3.2/bin$

ConcurrentHashMap misuse

Default constructor creates 16 segments which can consume excessive memory if lots of small Maps are constructed. Set the concurrency to 1 unless high contention is expected. http://ria101.wordpress.com/2011/12/12/concurrenthashmap-avoid-a-common-misuse/

svn revert cleanup

Revert all changes in a workspace svn revert -R . Remove all left over files with specific text in path after revert svn status | grep sl | awk '{print $2}' | xargs rm

Oracle JDBC ReadTimeout QueryTimeout

If a query exceeds the oracle.jdbc.ReadTimeout without receiving any data, an exception is thrown and the connection is terminated by the Oracle driver on the client. Unfortunately the session will still be queued on the database and continue to wait for locks, hold any current locks, and complete any DML/PL*SQL procedures that are pending on the server-side of the orphaned connection. If an application, on a another connection, due to ReadTimeout exception, retries DML/PL*SQL  which requires locks, those queries will queue behind the initial DML/PL*SQL. This will increase the workload exacerbating the situation. On applications with retries, this can be observed by querying the v$session table  or gv$session on RAC and noting new sessions started periodically based on the ReadTimeout interval. The sessions may often have the same SQL_ID and/or SQL_HASH_VALUE. If stmt.setQueryTimeout(Seconds) is issued and the statement exceeds the timeout, it will atte...

Locks held on Oracle for hours after sessions abnormally terminated by node failure

If a session holds db locks and is abnormally terminated (no fin/ack), the locks will persist until the db session is closed, typically around 2 hours and 12 minutes with default network tcp_keepalive settings. Abnormally terminated does not include CTRL C or kill -9. The ojdbc6 driver apparently has a shutdown hook thread that closes the connections in graceful shutdowns. The OS apparently closes the connections when processes are killed. Abnormally terminated might include power failure, firewall failures, Out Of Memory, OS/kernel crash, network connection failure, JBoss or other server node failure, etc. This was easily reproduced by creating a  process that connected to the db and updated a record but did not commit. While the first process was waiting, a second process with a contending update was started which blocked on the first update. The first clients network cable was then disconnected. The second clients transaction waited for approximately 2 hours until the databa...