Agent Remoting

The remoting code used to talk from agent to server (and server to agent for that matter) has layers. The lowest layer involves JBoss/Remoting. Sitting on top of JBoss/Remoting is the RHQ concept of "commands" (i.e. "command" objects are sent to remote endpoints via jboss/remoting). But even that requires a bit more code for most people to want to do, so on top of the command framework sits the "remote pojo" framework. This "remote pojo" framework is how we layer true "remote RPC" semantics in the agent-server communications. This page will talk about all these layers and how, in the end, "*AgentService" and "*ServerService" interfaces allow you to make normal Java calls actually perform remote RPC calls.

JBoss/Remoting

The communications layer used by the agent (and by the server as well) has JBoss/Remoting at its foundation. The RHQ comm layer builds on top of JBoss/Remoting - adding features like guaranteed delivery, throttling, and remote POJO invocations.

Commands/Command Responses/Command Services

Deep down in the RHQ comm layer (sitting on top of JBoss/Remoting) is the Command layer. All messages flowing into and out of the agent are Command objects. When a command reaches the remote server, it is processed by what is known as a Command Service which in turn responds with a Command Response object. This Command/CommandResponse framework is the basic building block upon which the other aspects of the communications framework is built upon.

Remote POJO Invocation Layer

Sitting higher up in the comm stack is the remote POJO invocation layer. This layer provides most of the communications API used by the rest of the subsystems. Developers write their POJOs as a normal Java object, remote them via the RHQ Remoting subsystem and they immediately become available for remote clients to call. Using the remote POJO invocation layer allows development to proceed without the developer having to know anything about the remoting layer while at the same time being given functionality such as guaranteed delivery, asynchronous invocations, failover, etc.

The RHQ Agent and RHQ Server both use this mechanism to access remote calls from each other. On the server, there are *ServerService interfaces, and on the agent there are *AgentService interfaces. More on this can be found in sections below.

Writing new commands / command services

Here's what you need to do to add new commands to the agent. Remember, these are low-level commands - usually, you won't have to do this if you are just adding new remote interfaces or new remote methods to existing remote interfaces. This section is here just for completeness.

• Create a Command
  • Extend AbstractCommand
  • Define the Command Type (the command name and version)
  • Define the Command parameters
  • Create methods that return strongly typed parameters
  • Define constructors that delegate to all superclass' constructors
  • EXAMPLE: see EchoCommand.java

• Create a Command Response
  • Extend AbstractCommandResponse
  • Create a method that returns the result object as a strongly typed object
  • Define constructors that delegate to all superclass' constructors
  • EXAMPLE see EchoCommandResponse.java

• Create a Command Service (the thing that executes the command on the server)
  • Extend CommandService or MultipleCommandService (if you want to more easily support multiple commands in a single service)
  • Override method getSupportedCommandTypes(), or getSupportedCommandTypeExecutors() if using MultipleCommandService, to define the commands this command service can execute
  • Implement the execute method (or write CommandExecutor objects if using MultipleCommandService)
  • EXAMPLE EchoCommandService

You don't technically need Commands and Command Responses - you can use GenericCommand/GenericCommandResponse - however, these are weakly typed and requires you to perform casting to actually process the commands in a more specific way (as opposed to generically processing commands, like the CmdlineClient does).

After these are defined, you add them to the agent. You can do this statically by adding your CommandService class name to communications.properties (the rhq.communications.command-services property). Or you can dynamically add command services at runtime. Requires rhq.communications.command-service-directory.allow-dynamic-discovery to be set to true in communications.properties (which it is in the default props file). Do add the services, you call ServiceContainer.addCommandService() passing in either a String (which is the classname of your service) or an Object (which is the command service instance).

RPC / remote POJOs

You can write POJOs that are remoteable. Use ServiceContainer.addRemotePojo() to make your pojo instance remoted. Use ClientRemotePojoInvoker.getRemotePojo() to get a proxy to the remoted POJO. This floats over the command infrastructure (over the RemotePojoInvocationCommand/CommandResponse), thus giving it all the functionality that the command stuff provides (like setting timeout for commands, guaranteed delivery, concurrent, asynchronous sending, synchronous sending, etc. etc.)

ServiceContainer

This singleton contains the server-side services. You can dynamically add new command services (so new commands can be processed) and you can add new network detection listeners so you can detect new servers (RHQ Agents or RHQ Servers both act as "comm servers") that come on and off line.

ClientCommandSender

This provides a client that can send commands to any remote server (RHQ Agent/RHQ Server). It can:

• send concurrent messages
• queue up commands when the maximum number of concurrent commands are in process of sending
• can send asynchronously (with a callback mechanism to tell you when its done) or synchronously.
• you can define what timeout to use - this is the time to wait before aborting a command. A command can define its own timeout or can just use the default timeout that is configured (agent.properties rhq.agent.client.command-timeout-msecs)
• you can enable throttling - the sender can be told that it can only send a maximum number of commands in a particular time period
• guaranteed delivery such that we persist the command to disk and only after we send it do we purge from disk

You can get an instance from the agent (AgentMain.getClientCommandSender() which sends the commands to the RHQ Server) or you can instantiate your own (requires you to know the remote endpoint or the RHQ Server or whatever remote server you want to send to).

When writing agent-side plugins, all of this remoting infrastructure and pojo registration is handled for you by the agent and its plugin container.

PersistentFifo

There is a class that persists data in a FIFO queue in a single file - the class is org.rhq.enterprise.communications.command.client.PersistentFifo. There are a bunch of tests that prove it works The sender object will use this to persist messages.

It persists byte[] data - but its API allows you to persist objects and take objects from the queue (it simply (de)serializes the object):

• void put(byte[])
• byte[] take()
• void putObject(Serializable object)
• Object takeObject()

You provide 3 things to the constructor - the File where the data is written, the maximum file size limit and a purge percentage. The maximum file size indicates how big we will allow the file to get before we need to shrink it somehow. If you put an entry on the FIFO, and that entry causes the file size to go over that max file size, this will trigger a purge. A purge will force the file to reduce in size such that it goes below the percentage of the max file size as defined by purge_percentage parameter (for example, if the maximum file size is 100KB and the purge percentage is 90, then a purge will shink the file down to no more than 90KB). A purge performs two things:

1. it compresses the file by removing all entries that have already been taken off the queue but still exist in the file
2. if compressing the file doesn't bring the file size down below the max, the purge will begin to remove entries from the queue (the oldest entries are removed first).

See the class javadocs for details on how this is implemented.

I think we can use this for both my needs (to persist commands for guaranteed delivery) and for the plugin container to use if it wants to squirrel away metric data to a local spool file, for example. You can squirrel away double[] arrays using putObject and takeObject for example.

This could be refactored to allow persisting the messages to a database, as opposed to a filesystem, if needed.

AgentService and ServerService Remote Interfaces

Sitting on top of the remote POJO infrastructure are the AgentService and ServerService interfaces. These are not found in the comm module, but rather they are extensions built into the RHQ Agent and RHQ Server codebase that build agent-specific and server-specific capabilities needed by the agent and server.

These are, essentially, the agent remote interfaces and server remote interfaces, as seen by each other. This is important to understand - these are not remote interfaces that other remote clients use (such as the CLI). These are strictly the contracts that the agent and server use to talk to each other.

For example:

org.rhq.core.clientapi.agent.configuration.ConfigurationAgentService is the RHQ Agent remote interface that the RHQ Server will use when it needs to talk to the agent's configuration subsystem.

org.rhq.core.clientapi.server.operation.OperationServerService is the RHQ Server remote interface that the RHQ Agent will use when it needs to talk to the server's operation subsystem..

You will notice that all agent service remote interfaces are located in the core/client-api module in subpackages under org.rhq.core.clientapi.agent. All server service remote interfaces are located in the same core/client-api module in subpackages under org.rhq.core.clientapi.server.

It is not possible to call a remote method from a plugin itself (i.e. a plugin developer cannot directly call a remote method on the RHQ server). It is important to remember that the plugin container is not always running inside an RHQ agent in a full blown RHQ environment. The plugin container can also be embedded in a managed resource itself (as a prime example, see Embeded Jopr which runs inside the JBossAS server and has no remote RHQ Server to talk to). You need to keep this in mind when writing plugin container code and plugin APIs. Plugins should never know anything about a "remote RHQ server" because in some runtime environments, a "remote RHQ Server" doesn't even exist. The plugins only talk to the plugin API (which under the covers is the plugin interface into the plugin container), that is it. This is a very important concept to understand - because if not, you will end up writing code that has wrong assumptions and won't work.

Adding remote interfaces to the RHQ Agent

Most of the time, you do not need to add an entirely new remote interface to the agent, because it is rare that you need to add a new subsystem. In most cases, you can simply add methods to existing remote interfaces (like ConfigurationAgentService) and that is all. Unless you have a good reason to add a new subsystem to the codebase, adding new methods to existing subsystem remote interfaces is what should be done.

However, if you need to add a new subsystem to the agent, you have to complete a few things.

1. First, you need to define a new *AgentService interface for the remote calls you want the agent to support. You place this interface in org.rhq.core.clientapi.agent (in the core/client-api module) and create a new implementation of it in the core/plugin-container module. See org.rhq.core.clientapi.agent.configuration.ConfigurationAgentService and org.rhq.core.pc.configuration.ConfigurationManager as an example.
2. Make sure your new remote implementation object extends org.rhq.core.pc.agent.AgentService. When a class extends this, the plugin container will inform the agent core that it needs to be "remoted" so a remote RHQ Server can call into it. Side note: when the plugin container is running inside an embedded server (like in the case of Embedded Jopr running inside a JBossAS app server), this is actually a no-op (i.e. the interface isn't actually remoted). However, if its running inside an RHQ Agent, it is remoted.
3. You will then have to hook them into the agent core by updating the plugin container initialization - see org.rhq.core.pc.PluginContainer.initialize(). The plugin container is where the actual implementation of your interface lives and thus it is here where you need to initialize your implementation class. This is how your new subsystem gets created.
4. Create getter methods in the PluginContainer so they can be retrieved properly. As an example, you should essentially do the same as what these methods do:
   • org.rhq.core.pc.PluginContainer.getConfigurationAgentService()
   • org.rhq.core.pc.PluginContainer.getConfigurationManager()
5. You need to add your new remote interface to the org.rhq.enterprise.server.agentclient.AgentClient interface and its implementation class org.rhq.enterprise.server.agentclient.impl.AgentClientImpl. This is the client object that the RHQ Server uses to talk to the agent. Adding your interface to it via a getter method will allow the server to talk to the agent's new remote interface you are creating. Don't forget to add the methods to the tests that also use this client class - it is easy to spot, once you change the client class, you'll see compile errors in your IDE to point you to where you need to add (e.g. TestAgentClient).

Be very very careful if your agent service impl thread needs to make a call back into the server. Assume that if you do this, a deadlock will occur! Unless you know what you are doing, do not make calls back into the server from the agent service. To avoid this, your remote method should utilize the returned object to send messages back to the server

.

Adding remote interfaces to the RHQ Server

Most of the time, you do not need to add an entirely new remote interface to the RHQ Server, because it is rare that you need to add a new subsystem. In most cases, you can simply add methods to existing remote interfaces (like OperationServerService) and that is all. Unless you have a good reason to add a new subsystem to the codebase, adding new methods to existing subsystem remote interfaces is what should be done.

However, if you need to add a new subsystem to the RHQ Server, you have to complete a few things.

1. First, you need to define a new *ServerService interface for the remote calls you want the server to support. You place this interface in org.rhq.core.clientapi.server (in the core/client-api module) and create a new implementation of it in the enterprise/server/jar module. See org.rhq.core.clientapi.server.operation.OperationServerService and org.rhq.enterprise.server.operation.OperationServerServiceImpl as an example.
2. The server remote interfaces are typically the ones that have Comm Annotations - specifically to define guaranteed delivery and limited concurrency. If you need that kind of functionality, use the comm annotations. Note that if you need to add limited concurrency, you have to add a new setting to the rhq-server.properties to define the new limited concurrency setting. If you do this, ensure you change all the things associated with that (e.g. org.rhq.enterprise.server.core.comm.ServerCommunicationsServiceMBean provides a way to change concurrency limits at runtime, which is used by the rhq-server plugin, so you'd need to modify the plugin to be able to use your new concurrency limit).
3. The new service needs to be added to the server-comm-configuration.xml file which is located at rhq/modules/enterprise/server/ear/src/main/resources/server-comm-configuration.xml. Find the entry with a key of rhq.communications.remote-pojos and append your service to the end of the list. Note that the list is comma-delimited, and the format of each entry is <service_impl>:<service_interface>. Be sure to specify the fully qualified class names.

That's really all you need. The ServerServiceImpl objects will make calls into the SLSB layer to actually perform the work that the agent needs to get done.

Be very very careful if your server service impl thread needs to make a call back into the agent. Assume that if you do this, a deadlock will occur! Unless you know what you are doing, do not make calls back into the agent from the server service. To avoid this, your remote method should utilize the returned object to send messages back to the agent

.

Comm Annotations

The core/comm-api module provides a few annotations that you can use to annotate your remote interfaces. These help you define additional behavior to your remote methods. See the org.rhq.core.communications.command.annotation package in the core/comm-api module for the different annotations you can use (these are javadoc'ed with additional information}.

You should read the Javadocs for more details, but here's a summary:

• @Asynchronous - allows you to define if a remote method is a "fire and forget" type method. The method call will not block waiting for it to return results (therefore, any method you annotate with this typically will have a "void" return type). You use this annotation, along with its attribute "guaranteedDelivery" to define methods that you require guaranteed delivery.

  For technical reasons, you should not define AgentService remote interface methods for guaranteed delivery. The RHQ Server should not be asked to persist messages for what is potentially thousands of remote agents.

• @LimitedConcurrency - this will denote methods that will be executed on a limited basis on the server-side of the client-server comm channel. This helps avoid message storms by limiting the number of method calls that can be processed at any one time.
• @Timeout - defines how long to give the remote method to finish executing before it is timed out and aborted
• @DisableSendThrottling - not used much, but this can ensure a message is sent a bit more timely (i.e. it turns off send throttling so the message will always be sent even if the send throttle would want to pause it). There is probably no reason to use this - so unless you have a very specific reason to use this, don't use it.

Remote Streams

This remote interface infrastructure supports the ability to do remote streaming. Your remote interfaces can pass InputStream and OutputStreams as parameters and return values. See the following as examples of this:

• InputStream org.rhq.enterprise.server.core.CoreServerServiceImpl.getPluginArchive(String)
• long org.rhq.core.clientapi.server.content.ContentServerService.downloadPackageBitsGivenResource(int, PackageDetailsKey, OutputStream)