Securing Server-Agent CommunicationsTypically, the RHQ Server and RHQ Agents talk to each other in the clear, meaning all communications traffic is unencrypted and no authentication is performed on either end. Many times, the environment in which you install your RHQ Servers and RHQ Agents does not warrant the extra setup time and runtime performance degradation you incur when enabling security on RHQ communications traffic (for example, if you already have a VPN and/or firewall protections in place that guard your RHQ Servers and RHQ Agents against intrusion). However, there are those that need or simply want the peace of mind of knowing their RHQ traffic is fully encrypted and authenticated. This section will describe the steps that you need to perform in order to fully secure the communications traffic between RHQ Servers and RHQ Agents.
Implications Of Not Using Secure CommunicationsRHQ does not secure the communications between the RHQ Server and RHQ Agent by default (out-of-box). Some issues that are of concern when running RHQ without secure communications are outlined below. You need to be aware of these issues before deciding to run RHQ with an unsecured communications channel between server and agent:
Running RHQ without securing the communications should only be done under the following circumstances, and only when you understand the full implications of doing so (as explained above):
RHQ and SSLRHQ utilizes SSL technology to perform both encryption and authentication. You can enable encryption (scrambling the data between server and agent to avoid someone eavesdropping on the traffic) and optionally enable authentication (which prohibits an intruder from attempting to spoof either a RHQ Server or RHQ Agent). It is recommended that you understand the basics of SSL and certificate-based security before attempting to secure your RHQ communications. EncryptionBy simply using a transport that uses SSL, you automatically get encryption. This means that when you configure your RHQ Server and RHQ Agent's communications layer, use an SSL-enabled transport to encrypt the traffic. SSL-enabled transports includes sslservlet and sslsocket. You do not have to worry about setting up certificates if you want to use SSL-enabled transports just for encryption. The RHQ Server ships with a certificate and the RHQ Agent will create a self-signed certificate if it needs one. Note that it is possible to just use SSL encryption without authentication. Some people may just wish to encrypt their RHQ traffic, without requiring the RHQ Servers and RHQ Agents authenticating each other. This setup, while less secure, is much easier to setup because it does not require creating and distributing trusted certificates. AuthenticationAuthentication via SSL requires that you distribute trusted certificates to your RHQ Servers and RHQ Agents. You must also configure those RHQ Servers and RHQ Agents to reject any messages coming from remote clients that do not match any of those trusted certificates. In order to support authentication, you must use SSL-enabled transports (which include sslservlet and sslsocket). You must also obtain trusted certificates for all your servers and agents and package those certificates in a set of keystore and truststore files. You can configure the RHQ Server to authenticate RHQ Agents, RHQ Agents to authenticate the RHQ Server or both. RHQ provides this flexibility in case you only want to authenticate in one direction but not another. However, when people feel the need for authentication, they will usually enable it in both directions. Authentication requires a bit more work to setup. Because true authentication requires a high degree of trust, you have to manually create and sign your certificates, create keystores and truststores that contain those certificates, then distribute your keystores and truststores in a highly secure manner to all your RHQ Servers and RHQ Agents. This may mean going as far as physically hand-delivering your trusted certificates via CD and copying the certificates from the CD to all the computers hosting the RHQ Servers and RHQ Agents. Many times it is not as highly paranoid as that - however, at some point along the way, you have to place trust in whatever certificates you are using and distributing. Setting Up Secure RHQ CommunicationsThe following are the steps necessary to set up secure communications in RHQ. Remember that you have the option for encryption-only or encryption-with-authentication. If you choose the former, you do not need to perform any of the steps dealing with the creation, packaging and distribution of certificates and keystores/truststores. If you wish to have full security with encryption-with-authentication, follow all the steps listed below. If you are setting up with encryption-with-authentication, these steps assume you want authentication in both directions (that is, the agents will need to authenticate with the server and the server will need to authenticate with the agents). The RHQ low-level communications layer is based on JBoss/Remoting and uses what is known as a transport to ship messages back and forth between agents and servers. A transport can be either unencrypted (like the raw socket transport or servlet) or it can be encrypted (like sslsocket or sslservlet).
The servlet-based transports, servlet and sslservlet, are HTTP and HTTPS transports respectively. They are "servlet"-based because the HTTP/HTTPS traffic is routed through a servlet running in the Tomcat server hosted within the RHQ Server. We use these servlet based transports because they leverage the highly performant Tomcat connector infrastructure with no need for additional thread pooling to accept incoming agent requests since Tomcat will handle the requests. Using these servlet-based transports means that the configuration you set up here for agent communications are the same settings that take effect for user requests coming into the GUI via a browser.
For the following steps, we are going to configure the server-to-agent channel to use the sslsocket transport and the agent-to-server channel to use the sslservlet transport.
Step 1 - Use SSL Transport To Enable EncryptionThe first thing you need to do is tell the RHQ Servers and RHQ Agents to use SSL when talking to each other. RHQ Server Instructions
RHQ Agent Instructions
At this point, you have now configured your RHQ Servers and RHQ Agents to encrypt their messages to each other via SSL. You can be assured that no one can effectively eavesdrop on the communications between them. However, if you wish to strengthen the security of the network traffic even further, continue on to the next series of steps which will enable certificate-based authentication between your RHQ Servers and RHQ Agents. Step 2 - Prepare Your SSL CertificatesIf you wish to have your RHQ Servers and RHQ Agents authenticate one another, you need something that "identifies" each one of them. SSL requires digital certificates for this purpose. If your company or organization can request and receive officially signed certificates from a trusted CA, you will need to obtain one certificate for each of your RHQ Servers and RHQ Agents that you plan on deploying in your RHQ environment. If you already have the certificates given to you from your CA, you must place each of them in their own keystore file and combine all of them into a single truststore file. Otherwise, please follow the instructions to create your own certificates. The purpose of these instructions is to generate a keystore file and a truststore file for each RHQ Server and RHQ Agent. Each keystore file will contain a single self-signed certificate that belongs to one of the RHQ Server or RHQ Agent entities. Each truststore file contains all the certificates belonging to every RHQ Server and RHQ Agent.
The important part is to make sure you set the Common Name (CN) of the Distinguished Name (the -dname option) to the correct address where this keystore is to be installed. That is because as part of the SSL handshake, a remote client will attempt to verify that the issuer of the certificate (as listed in the CN) is the same name as where the certificate actually came from. Now that we have generated a self-signed certificate for this RHQ Agent and stored it in a file named "myhost-keystore.dat", we can store that away for now and continue generating certificates/keystores for the rest of the machines until we have one keystore file for each and every machine that will host a RHQ Server or RHQ Agent. It is best to name the keystores so you remember which keystore file belongs to which machine (hence why, in the example above, the hostname was part of the filename). Same holds true with the alias names.
Step 3 - Distribute Your Keystores and TruststoresAt this point, you have created a set of keystore files (one for each RHQ Server and RHQ Agent in your RHQ environment) and a single truststore file (a duplicate copy is to be given to each RHQ Server and RHQ Agent). You must now distribute those files to all the machines where your RHQ Servers and RHQ Agents live. You must do so in a secure fashion and ensure that no one can steal, intercept or otherwise manipulate your keystore/truststore files. You must also make sure that you distribute the keystore files to the host machines that match the certificates' CN host addresses. If you mix them up and, for example, put the "myhost" keystore file on the "anotherhost.mycorp.com" machine, the SSL communications will fail for the RHQ Server or RHQ Agent running on "anotherhost". RHQ Server InstructionsEach RHQ Server distribution has a JBossAS instance within it. That JBossAS has a conf directory that you can use for the location to store the server's keystore/truststore files (but technically, you can put them anywhere that the server can access them. Please ensure you remember the location because you will need it for the next step). For each RHQ Server, take its keystore file (make sure the keystore file has the appropriate CN value that matches the RHQ Server's hostname) and store it in $RHQ_SERVER_HOME/jbossas/server/default/conf under the name keystore.dat. Make a copy of your truststore file and place it in that same directory under the name truststore.dat. RHQ Agent InstructionsEach RHQ Agent distribution has a /conf directory. It is the logical choice to store the agent's keystore/truststore files. (note: putting them here makes them safe when performing agent auto-updates - agents will retain all keystore/truststore files that are found in the /conf and /data directory). For each RHQ Agent, take its keystore file (make sure the keystore file has the appropriate CN value that matches the RHQ Agent's hostname) and store it in the agent's /conf directory. Make a copy of your truststore file and place it in the agent's /conf directory as well. Step 4 - Tell The RHQ Servers and Agents About Their Keystores/TruststoresNow you have to tell your RHQ Servers and RHQ Agents where your keystore and truststore files are in addition to providing other information about those files so they can be read properly. After completing this step, your RHQ Servers and RHQ Agents will be able to successfully authenticate themselves to each other. RHQ Server Instructions
RHQ Agent Instructions
At this point, you have now configured your RHQ Servers and RHQ Agents to both encrypt their messages to each other and to authenticate each other via SSL. You now can be assured that no one can effectively eavesdrop on your RHQ communications nor can an infiltrator attempt to spoof itself as a bogus RHQ Server or RHQ Agent. Step 5 - Test Your SetupOnce you are done with the preceding steps, you can finally restart your RHQ Servers and RHQ Agents. They should begin to talk to each other normally - if you have done everything correctly, you will not notice anything different! If you want to confirm that they really are using SSL authentication, simply remove a keystore or truststore file from either a RHQ Server or RHQ Agent and you should begin to notice errors appear in their log files. Removing a keystore file or truststore file from a RHQ Agent will prohibit that agent from being able to send and receive messages to/from the RHQ Server - which you can confirm by looking at the agent log file and looking for error messages. After you've finished testing, make sure you remember to restore the keystore/truststore files. You can test the SSL authentication by creating another keystore for one of your RHQ Agents, replace that keystore with the original keystore and try to see if that RHQ Agent can talk to the RHQ Server. Because this new keystore has a certificate that does not exist in the RHQ Server's truststore, the RHQ Server will no longer trust that agent and will reject its messages. In effect, you simulated an infiltrator trying to spoof a RHQ Agent and the RHQ Server detected this security breach. Setting Up Server-Side sslsocket TransportAs mentioned previously, if you use the sslservlet transport, messages from the agent to the server are routed through Tomcat, the same as when your users make browser requests to the server GUI. If:
then you will also require your users' browsers to have certificates installed and those user certificates must be placed in your server-side truststore file; otherwise, your users that try to access the server GUI over https: will find that Tomcat will reject their requests due to missing certificates, even if the user can authenticate themselves using their RHQ username and password. In highly secure environments, this may not be a problem because users might already have certificates assigned to them and installed in their browser and you might be able to get or build a truststore that contains all of your users' certificates. However, under most situations, this is not desireable. Users normally do not have their own certificates installed in their browsers and even if they did, you probably cannot obtain their certificates ahead of time to place them in the server's truststore. The users will authenticate themselves to the server GUI via their own RHQ username and password. So the question becomes, how can you require agents to authenticate themselves with certificates without requiring users to do so? The answer lies in the ability of RHQ to use a different transport. Rather than use sslservlet, we can tell the RHQ Server to use sslsocket instead. This will enable the RHQ Server to open a special server-side socket that you designate that will be used to accept agent requests. This special socket will circumvent Tomcat and the Tomcat configuration - it will have its own SSL security configuration, allowing you to configure Tomcat to be less strict in the requests it accepts.
All of the steps covered in previous sections are still valid. The only difference in being able to use sslsocket versus sslservlet is a few configuration setting changes. These differences will be explained below. sslsocket RHQ Server SettingsIn general, the security settings with the names that start with rhq.server.tomcat.security do not need to be changed and should be left as they were when the server was initially installed; instead, you use the rhq.communications.connector.security settings: rhq.communications.connector.transport=sslsocket
rhq.communications.connector.bind-address=
rhq.communications.connector.bind-port=55555
rhq.communications.connector.transport-params=
rhq.communications.connector.security.secure-socket-protocol=TLS
rhq.communications.connector.security.keystore.file=${jboss.server.home.dir}/conf/keystore.dat
rhq.communications.connector.security.keystore.algorithm=SunX509
rhq.communications.connector.security.keystore.type=JKS
rhq.communications.connector.security.keystore.password=jonpassword
rhq.communications.connector.security.keystore.key-password=jonpassword
rhq.communications.connector.security.keystore.alias=myhost
rhq.communications.connector.security.truststore.file=${jboss.server.home.dir}/conf/truststore.dat
rhq.communications.connector.security.truststore.algorithm=SunX509
rhq.communications.connector.security.truststore.type=JKS
rhq.communications.connector.security.truststore.password=jonpassword
rhq.communications.connector.security.client-auth-mode=true
Here the transport is now sslsocket. Because we are using sslsocket transport, you need to indicate which port to bind to (this is the port the server will listen to when receiving agent requests - in this example we used port 55555 but you can use any free port). The bind-address can still be left as-is, it will pick up a default for the server. You can specify a bind-address if you want to explicitly tell the server what to bind to. The transport-params can be any valid JBoss/Remoting transport parameters - this can be left blank. See the JBoss/Remoting documentation for more information on what you can do with these transport parameters. For the security settings, all have the same values as before except their names start with rhq.communications.connector.security, not rhq.server.tomcat.security.
sslsocket RHQ Agent SettingsVery little has to change from the previous instructions to get the agent to talk to the server over the sslsocket protocol. The only difference is that you have to tell the agent to use the sslsocket transport, what the new port is and what the server's transport parameters are:
Troubleshooting Secure Communications SetupCorrect configuration for secure communications can be difficult at times because problems in any of the following areas can halt successful communications between server and agent and frustrate the process:
If you've already gone through the detailed instructions above for setting up secure RHQ communications but are still having difficulties it is helpful to double check with the following steps.
For simplicity we'll assume that you have addressed the following fundamental setup instructions. You should verify the certificates with keytool if haven't already:
Example of relevant RHQ Server Shared default settings: <rhq-server-install-dir>/bin/rhq-server.properties # RHQ Server's remote endpoint for agents to talk to
# bind-address and bind-port are derived from the HA server definition,
# if you set the address/port here, they will override the HA server definition found in the database
rhq.communications.connector.transport=servlet
rhq.communications.connector.bind-address=
rhq.communications.connector.bind-port=
rhq.communications.connector.transport-params=/jboss-remoting-servlet-invoker/ServerInvokerServlet
# Server-side SSL Security Configuration for HTTPS thru Tomcat
# These are used for browser https: access and for incoming messages from agents over sslservlet transport
# [you cannot use ${x} variables - see https://jira.jboss.org/jira/browse/JBWEB-74]
rhq.server.tomcat.security.client-auth-mode=want
rhq.server.tomcat.security.secure-socket-protocol=TLS
rhq.server.tomcat.security.algorithm=SunX509
rhq.server.tomcat.security.keystore.alias=RHQ
rhq.server.tomcat.security.keystore.file=conf/rhq.keystore
rhq.server.tomcat.security.keystore.password=RHQManagement
rhq.server.tomcat.security.keystore.type=JKS
rhq.server.tomcat.security.truststore.file=conf/rhq.truststore
rhq.server.tomcat.security.truststore.password=RHQManagement
rhq.server.tomcat.security.truststore.type=JKS
# Server-side SSL Security Configuration (for incoming messages from agents)
# These are used when secure transports other than sslservlet are used
rhq.communications.connector.security.secure-socket-protocol=TLS
rhq.communications.connector.security.keystore.file=${jboss.server.home.dir}/conf/rhq.keystore
rhq.communications.connector.security.keystore.algorithm=SunX509
rhq.communications.connector.security.keystore.type=JKS
rhq.communications.connector.security.keystore.password=RHQManagement
rhq.communications.connector.security.keystore.key-password=RHQManagement
rhq.communications.connector.security.keystore.alias=RHQ
rhq.communications.connector.security.truststore.file=${jboss.server.home.dir}/conf/rhq.truststore
rhq.communications.connector.security.truststore.algorithm=SunX509
rhq.communications.connector.security.truststore.type=JKS
rhq.communications.connector.security.truststore.password=RHQManagement
rhq.communications.connector.security.client-auth-mode=none
# Client-side SSL Security Configuration (for outgoing messages to agents)
rhq.server.client.security.secure-socket-protocol=TLS
rhq.server.client.security.keystore.file=${jboss.server.home.dir}/conf/rhq.keystore
rhq.server.client.security.keystore.algorithm=SunX509
rhq.server.client.security.keystore.type=JKS
rhq.server.client.security.keystore.password=RHQManagement
rhq.server.client.security.keystore.key-password=RHQManagement
rhq.server.client.security.keystore.alias=RHQ
rhq.server.client.security.truststore.file=${jboss.server.home.dir}/conf/rhq.truststore
rhq.server.client.security.truststore.algorithm=SunX509
rhq.server.client.security.truststore.type=JKS
rhq.server.client.security.truststore.password=RHQManagement
rhq.server.client.security.server-auth-mode-enabled=false
Examples of relevant RHQ Agent Shared default settings: <rhq-agent-install-dir>/conf/agent-configuration.xml <entry key="rhq.agent.server.transport" value="servlet" />
<entry key="rhq.agent.server.bind-port" value="7080" />
<!--
<entry key="rhq.agent.server.bind-address" value="127.0.0.1" />
-->
<entry key="rhq.agent.server.transport-params" value="/jboss-remoting-servlet-invoker/ServerInvokerServlet" />
<entry key="rhq.agent.server.alias" value="rhqserver" />
<!--
Notice about bind-address and bind-port:
If you use transport param "serverBindAddress", that will
actually be the address the agent will use to bind its
socket and the rhq.communications.connector.bind-address
will be the public address the RHQ Server will use to
talk to the agent.
If you use transport param "serverBindPort", that will
actually be the port the agent will use for its server
socket and the rhq.communications.connector.bind-port
will be the public port the RHQ Server will use to
talk to the agent.
-->
<entry key="rhq.communications.connector.rhqtype" value="agent" />
<entry key="rhq.communications.connector.transport" value="socket" />
<entry key="rhq.communications.connector.bind-port" value="16163" />
<!--
<entry key="rhq.communications.connector.bind-address" value="127.0.0.1" />
<entry key="rhq.communications.connector.transport-params" value="serverBindAddress=127.0.0.1& \
serverBindPort=16163&numAcceptThreads=3&maxPoolSize=303&clientMaxPoolSize=304& \
socketTimeout=60000&enableTcpNoDelay=true&backlog=200" />
<entry key="rhq.communications.connector.lease-period" value="5000" />
-->
Unencrypted/unsecured Communication Setup
SSL Communication(Encryption only) Setup
SSL Communication(Encryption with Client Cert Authentication) Setup
Still need more SSL debugging informationIf you have double checked all of your settings, certificates and keystore passwords and you do not see any helpful log messages within the server or agent logs then you can additionally enable verbose SSL communication messaging within the agent to attempt to get more helpful communication information:
|
