Is cPanel compatible with mod security (mod_security) ?

Yes, you can install mod_security in WHM under Plugins.

More information about configuring mod_security can be found at: http://modsecurity.org/