Fault Tolerance in High Volume Distributed Systems

Netflix Blog

DependencyCommand wraps network bound commands

  • has fallback logic
  • executes in separate thread pools
  • uses try/acquire
  • has timeouts
  • latency feeds back to circuit breakers

Comments

Post a Comment

Popular posts from this blog

Java InetAddress isReachable ICMP (Ping)

It doesn't do ICMP on Windows, it opens TCP echo on port 7. On linux/unix you have to set the suid bit on Java executables to allow it to use ICMP. So that article didn't clearly explain the whole story. When the executable bit is set if a user has permissions to run the program the executable runs at permissions of the owner. If like mine, your java was owned by account with non-root privileges, it still won't be able to access ICMP. You have to set suid bit after changing ownership as the latter clears the suid bit. sudo chown -R root:root sudo chmod 4755 /bin/java Never did get this to work because once the suid bit was set, encountered the following error: libjli.so file not found. I believe this might have to do with suid bit not being set on LD_LIBRARY_PATH references and/or shared libs because it might present a security issue. After discussing with associates realized going this route is probably a bad idea. This would cause all Java programs to run as root presenti...
Read more

Oracle JDBC ReadTimeout QueryTimeout

If a query exceeds the oracle.jdbc.ReadTimeout without receiving any data, an exception is thrown and the connection is terminated by the Oracle driver on the client. Unfortunately the session will still be queued on the database and continue to wait for locks, hold any current locks, and complete any DML/PL*SQL procedures that are pending on the server-side of the orphaned connection. If an application, on a another connection, due to ReadTimeout exception, retries DML/PL*SQL  which requires locks, those queries will queue behind the initial DML/PL*SQL. This will increase the workload exacerbating the situation. On applications with retries, this can be observed by querying the v$session table  or gv$session on RAC and noting new sessions started periodically based on the ReadTimeout interval. The sessions may often have the same SQL_ID and/or SQL_HASH_VALUE. If stmt.setQueryTimeout(Seconds) is issued and the statement exceeds the timeout, it will atte...
Read more

ActiveMQ https

ActiveMQ supports tunneling JMS over https to transparently traverse firewalls, but the  documentation  doesn't indicate how to configure it and relevant examples were lacking via google. After hacking for half a day I finally stumbled upon the simple solution. Keep in mind that you will want to create you own keys and certificate. For testing I just used the ones provided in the activemq/conf dir. Simply configure an https transport connector in conf/activemq.xml and pass the keystore, password, and truststore to activemq when started. conf/activemq.xml <amq:transportconnector uri="https://0.0.0.0:61684"> activemq start parameters -Djavax.net.ssl.keyStore=/path/activemq/conf/broker.ks -Djavax.net.ssl.keyStorePassword=password -Djavax.net.ssl.trustStore=/path/activemq/conf/broker.ts import self-signed cert into client jdk keystore cd /path/jdk1.7.0_15/jre/lib/security keytool -import -trustcacerts -file /path/activemq/conf/broker-l...
Read more