A few years ago an NTP issue came to light that caused a lot of damage. Cloudflare did a good writeup on this if you want to see the details. But in a nutshell, if there is a request which can be sent which causes a larger response, you have amplification. In the NTP case, a small packet requesting the peer list would get a much larger response. Coupled with being UDP and a lot of providers not implementing BCP38, meaning I can generate a request from your IP, and the large response goes to you, and you have a problem.
OK what does this have to do with cloud you ask? Well, lets look at logging. A lot of people use ‘logging as a service’ (stackdriver, elasticsearch, …). It can be managed by your cloud provider (Google, Microsoft, …), or by a 3rd party. But, well, you log a lot of stuff in (moderatly) abnormal circumstances.
Now lets look at a particular tool. ‘Tranquility‘ (this observation seems true for nearly any Java program as far as I have observed). When something happens, it logs a stack trace. As an end-user? useless. Lets look at one below. When someone connects to my service without knowing my credentials, they eventually time out and leave this bomb. Its 9747 bytes. From a single connect. This is in fact worse than the above NTP issue.
But it gets worse. Each of these lines is bundled up into JSON, with a timestamp, some fields around originating host, etc. It turns out by the time this lands on the wire on the way out to my log provider its more like 20KB. That’s right, a hundred bytes or so of inbound SYN and ACK cause this. Now I know what you are thinking, what endpoint of Don’s can I try this on? Well its ‘http://127.0.0.1‘. Go ahead.
How long before someone finds and attacks this vector? Hmm.
2019-02-03 22:08:00,959 [qtp1247938090-34] WARN c.m.t.server.http.TranquilityServlet - Server error serving request to http://tranquility.druid.agilicus.ca:443/v1/post/logs java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039) ~[na:1.8.0_181] at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328) ~[na:1.8.0_181] at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277) ~[na:1.8.0_181] at com.twitter.util.Promise.ready(Promise.scala:667) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0] at com.twitter.util.Promise.result(Promise.scala:673) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0] at com.twitter.util.Await$$anonfun$result$1.apply(Awaitable.scala:151) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0] at com.twitter.concurrent.LocalScheduler$Activation.blocking(Scheduler.scala:220) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0] at com.twitter.concurrent.LocalScheduler.blocking(Scheduler.scala:285) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0] at com.twitter.concurrent.Scheduler$.blocking(Scheduler.scala:115) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0] at com.twitter.util.Await$.result(Awaitable.scala:151) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0] at com.twitter.util.Await$.result(Awaitable.scala:140) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0] at com.metamx.tranquility.tranquilizer.Tranquilizer.flush(Tranquilizer.scala:243) ~[io.druid.tranquility-core-0.8.3.jar:0.8.3] at com.metamx.tranquility.server.http.TranquilityServlet$$anonfun$doSend$3.apply(TranquilityServlet.scala:204) ~[io.druid.tranquility-server-0.8.3.jar:0.8.3] at com.metamx.tranquility.server.http.TranquilityServlet$$anonfun$doSend$3.apply(TranquilityServlet.scala:204) ~[io.druid.tranquility-server-0.8.3.jar:0.8.3] at scala.collection.mutable.HashMap$$anon$2$$anonfun$foreach$3.apply(HashMap.scala:108) ~[org.scala-lang.scala-library-2.11.8.jar:na] at scala.collection.mutable.HashMap$$anon$2$$anonfun$foreach$3.apply(HashMap.scala:108) ~[org.scala-lang.scala-library-2.11.8.jar:na] at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230) ~[org.scala-lang.scala-library-2.11.8.jar:na] at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) ~[org.scala-lang.scala-library-2.11.8.jar:na] at scala.collection.mutable.HashMap$$anon$2.foreach(HashMap.scala:108) ~[org.scala-lang.scala-library-2.11.8.jar:na] at com.metamx.tranquility.server.http.TranquilityServlet.doSend(TranquilityServlet.scala:204) ~[io.druid.tranquility-server-0.8.3.jar:0.8.3] at com.metamx.tranquility.server.http.TranquilityServlet.com$metamx$tranquility$server$http$TranquilityServlet$$doV1Post(TranquilityServlet.scala:141) ~[io.druid.tranquility-server-0.8.3.jar:0.8.3] at com.metamx.tranquility.server.http.TranquilityServlet$$anonfun$4.apply(TranquilityServlet.scala:87) ~[io.druid.tranquility-server-0.8.3.jar:0.8.3] at com.metamx.tranquility.server.http.TranquilityServlet$$anonfun$4.apply(TranquilityServlet.scala:85) ~[io.druid.tranquility-server-0.8.3.jar:0.8.3] at org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$liftAction(ScalatraBase.scala:270) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala:265) ~[org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala:265) ~[org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.ScalatraBase$class.withRouteMultiParams(ScalatraBase.scala:341) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.ScalatraServlet.withRouteMultiParams(ScalatraServlet.scala:49) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.ScalatraBase$class.invoke(ScalatraBase.scala:264) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.ScalatraServlet.invoke(ScalatraServlet.scala:49) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$apply$8.apply(ScalatraBase.scala:240) ~[org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$apply$8.apply(ScalatraBase.scala:238) ~[org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at scala.Option.flatMap(Option.scala:171) ~[org.scala-lang.scala-library-2.11.8.jar:na] at org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply(ScalatraBase.scala:238) ~[org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply(ScalatraBase.scala:237) ~[org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at scala.collection.immutable.Stream.flatMap(Stream.scala:489) ~[org.scala-lang.scala-library-2.11.8.jar:na] at org.scalatra.ScalatraBase$class.runRoutes(ScalatraBase.scala:237) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.ScalatraServlet.runRoutes(ScalatraServlet.scala:49) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.ScalatraBase$class.runActions$1(ScalatraBase.scala:163) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply$mcV$sp(ScalatraBase.scala:175) ~[org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply(ScalatraBase.scala:175) ~[org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply(ScalatraBase.scala:175) ~[org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$cradleHalt(ScalatraBase.scala:193) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.ScalatraBase$class.executeRoutes(ScalatraBase.scala:175) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.ScalatraServlet.executeRoutes(ScalatraServlet.scala:49) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.ScalatraBase$$anonfun$handle$1.apply$mcV$sp(ScalatraBase.scala:113) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.ScalatraBase$$anonfun$handle$1.apply(ScalatraBase.scala:113) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.ScalatraBase$$anonfun$handle$1.apply(ScalatraBase.scala:113) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) [org.scala-lang.scala-library-2.11.8.jar:na] at org.scalatra.DynamicScope$class.withResponse(DynamicScope.scala:80) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.ScalatraServlet.withResponse(ScalatraServlet.scala:49) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.DynamicScope$$anonfun$withRequestResponse$1.apply(DynamicScope.scala:60) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) [org.scala-lang.scala-library-2.11.8.jar:na] at org.scalatra.DynamicScope$class.withRequest(DynamicScope.scala:71) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.ScalatraServlet.withRequest(ScalatraServlet.scala:49) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.DynamicScope$class.withRequestResponse(DynamicScope.scala:59) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.ScalatraServlet.withRequestResponse(ScalatraServlet.scala:49) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.ScalatraBase$class.handle(ScalatraBase.scala:111) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.ScalatraServlet.org$scalatra$servlet$ServletBase$$super$handle(ScalatraServlet.scala:49) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.servlet.ServletBase$class.handle(ServletBase.scala:43) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.ScalatraServlet.handle(ScalatraServlet.scala:49) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at org.scalatra.ScalatraServlet.service(ScalatraServlet.scala:54) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet.javax.servlet-api-3.1.0.jar:3.1.0] at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:800) [org.eclipse.jetty.jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587) [org.eclipse.jetty.jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:517) [org.eclipse.jetty.jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [org.eclipse.jetty.jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [org.eclipse.jetty.jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112] at org.eclipse.jetty.server.Server.handle(Server.java:497) [org.eclipse.jetty.jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) [org.eclipse.jetty.jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248) [org.eclipse.jetty.jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112] at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [org.eclipse.jetty.jetty-io-9.2.5.v20141112.jar:9.2.5.v20141112] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:620) [org.eclipse.jetty.jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112] at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:540) [org.eclipse.jetty.jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
Leave a Reply