We present a novel mechanism for detecting and protecting structured overlay networks against non-conforming (abnormal) behavior of other participating nodes. We use a lightweight distributed detection mechanism that exploits inherent structural invariants of DHTs to ferret out anomalous flow behavior. To prevent identity spoofing leading to Sybil attacks, neighbor identities are established with pair-wise keys, which do not require an authentication infrastructure. Upon detection, a Pushback-like protocol is invoked to notify the predecessor whence the offending traffic is arriving. Recursive applications of the protocol can identify and isolate the offending node.

We evaluate our mechanism's ability to detect attackers via simulation within a DHT network. The results show that our system can detect a simple attacker whose attack traffic deviates by as little as 5\% from average traffic. We also demonstrate the resiliency of our mechanism against coordinated distributed flooding attacks that involve up to 15\% of overlay nodes. We measure the effectiveness with which our approach identifies the offending node(s) and squelches the attacks. The detection and containment mechanisms presented show that overlays can protect themselves from insider DoS attacks, eliminating an important roadblock to their deployment.