Reinforcement learning (RL) has remarkably enabled multi-agent systems to achieve policies that can deliver stateof-the art control in complex learning environments like playing Atari games or solving Go game with infinitely broad state-action horizon [1]. RL approaches are being widely used for training agents to learn control over complex gait maneuvers (like crawling, walking or running) solely based on their own learning experiences and corresponding rewards associated with that experience. Well established RL frameworks allow agents to learn through trial and error and stay motivated by improving performance over selecting better actions iteratively at next time step to yield high reward. Most of the RL methods assume that agents are at freedom to choose any actions that satisfies the condition of achieving maximum expected reward in return -hence reward being the major performance factor overall in action selection [2]. Reward engineering is considered to be one of the significant challenges for learning an action-selection policy for real-world problem scenario because, in real-world behavior; rewards are often delayed and are also decayed with time [3], [4]. In multi-agent setting, policy search algorithms need to be modeled with generic behavioral rewards such that a policy must not go adverse to any agent and should also suffice for overall shared goals of all agents -not just a single agent. One of the fundamental challenges for multi-agent robots is learn an equivalently beneficial policy for all agents spawned in complex dynamic environments with sparse or delayed feedback behavior [3]. Learning in such setting requires completely observable representation of model for efficient exploration, or a well defined heuristic for action-selection to narrow down action space; however modeling fully observable environment is mostly unattainable or requires exceedingly complicated computations. Policy search algorithms have been in use for a long time to address action-selection problem by learning a function π(s) (or a rule) that can return a decision to choose an action given available information of the current state s given s ∈ S, even with partial or no model representations. The problem with the concept of a policy is that it refers to any method for determining an action given a state, and as a result it covers a wide range of algorithmic strategies, each suited to different problems with different computational requirements [5]. Current policy search algorithms also provide notable performance for learning an optimal policy for continuous domains. Such methods commonly perform non-linear function approximation coupled with off-the-shelf policy gradient algorithms to learn state-action transitions over high-dimensional continuous spaces [6]- [10].
Although these methods significantly improve convergence and give notable control capabilities in highdimensional continuous spaces; most of these methods can only perform under full model observability and deterministic beliefs of optimality irrespective of notorious behavior of sparse rewards and also exploration is mostly engineered heuristically (ex: by injecting noise) [11]- [13] or is set by initializing a stochastic policy with high entropy [13]. Agents that possess intricate geometrical structure (like bioinspired robots or soft robots) cannot be explicitly modeled due to their redundant kinematic properties and due to larger pool of possible sub-optimal solutions that make it difficult to converge towards most optimal solution [14] [15]. Secondly, agents with complex structures tend to have exceedingly convoluted action spaces due to redundant joints causing curse of dimensionality issues; any sub-optimal action selection at low level will embrace overall deteriorated action selection. Agents with complex reward behavior due to physical constraints can only receive sparse rewards because reward distribution for such agents (ex. bio-inspired manipulator) is based on expectation of reward as function of entire action-selection transitions -and action space is continuous and vectorized sequence of connected acts if an agent has connected structure (ex:bio-inspired robots) [16], [17]. Since most traditional algorithms with "vanilla" policy gradient methods are conceived entirely in context of Markov Decision Processes (MDP), these methods significantly enjoy high throughput over control but at a cost of exhaustive complexity in terms of extended exploration requirement and longer learning periods [18]. Also such methods easily overlook abstract simplicites available at temporal scales [19].
We present an approach that learns a policy in highdimensional continuous horizon by scaling down the dimensions into identical sub-dimensions while keeping the continuous horizon intact, and by ranking the learning problem into local and global learning layers such that global learning layer is eventually engineered quickly with positive learning bias of its local layers' knowledge. The approach is reminiscent to human nature of doing sequential but lengthy tasks in daily routine. For the sake of simplicity; take an example of opening a drawer-a person first needs to maneuver the arm forward (like a servo maneuver) towards the drawer handle, grasp it (consider grasping as already learned skill), and move the arm backwards against the drawer. The backward maneuver (servo maneuver) actually shares a similar problem model as forward movement; but the immediate reward for backward joint movement might get negative if drawn by forward maneuver policy. However the global or overall reward should still be optimal since the main goal was to open a drawer. Our constructive policy approach ranks such identical tasks as local tasksgenerating a pool of local policies. Next step involves to learn deploying optimal combination of policies by sampling them with optimal policy parameters. While learning global policy we modify the single agent into scaled down hierarchy shadowing apprenticeship learning methodology that allows the system to accept advice or bias from multiple local apprentices operating at lower layers [10]. The contribution of our study is build upon decomposition of high-dimensional spaces into connected multi-agent space scenario that offer practical benefits for real-world robot models as well. We demonstrate our method to control remarkably complex robotic arm design with 12 DOFs -once trained, the arm follows continuous trajectories in continuous domain.
Policy learning for agents operating in continuous horizons and sparse reward environments has been studied as temporal abstraction problem that inherently constitutes a Semi-MDP (SMDP) learning model such that an agent has a policy with parameters θ ∃ π θ :S × A → [0, 1], and follows a stochastic termination condition β:S + → [0, 1]. The agent takes an action a t ∈ A and continues to evaluate towards a t+1 with Markovian property until it terminates stochastically with probability distribution of β. When the evaluation is terminated the agent can stochastically choose to timeout and restart with newer policy parameter π θ * or continue determining actions with same parameters [18]. At higherlevel, an agent can choose actions that draw varying amount of time to finish (randomly distributed time) and any sojourn time in given state is actually a continuous random variable with distribution depending on states [18], [20], [21].
With SMDPs, the expected recurrent time-average outcomes can be embedded under a stationary policy with recurrent states and can be generalized for primitive actions -furthermore those primitive actions are extended over temporal course of actions. This learning method is used for tasks that can be designed with shared meta-controls [22] and integrated temporal abstractions (options framework, Sutton [18]). For high dimensional or infinite spaces, approximation can be maximum entropy value approximation over established distribution models (ex: Boltzmann distribution or Thompson sampling) and choose actions with respect to randomly drawn belief that instantaneously is self-corrected over time steps following Bayesian control rule for calculating expected rewards. The rewards can be weighted for a cost utility function that quantifies overall desirability of corresponding belief [7], [23], [24]. Hierarchical or layered reinforcement learning is broadly studied topic in field of artificial intelligence. It is widely used for learning and opti-Authorized licensed use limited to: University of Central Florida. Downloaded on September 23,2020 at 03:57:13 UTC from IEEE Xplore. Restrictions apply.
Reusing local policy with learned global parameters. Colors show sub-agent intercepts operating under same local policy with different parameters mizing policies for environments that cannot be modeled into discrete representations and are also infinitely huge in terms of model dimensions. The crucial part in modeling policy is adapting actions based on dynamic primitives. Policy based methods such as splines gradients or often called vanilla policy gradients algorithms are popular in discrete space behavioral policy modeling due to their trajectory centric representation of actions. These methods are widely seen to solve complex tasks like bi-pedal robot walking [25] or learning primitive motor angles to control joints [26]; also these vanilla policy gradient methods are relatively easy to implement and adapt in real-world as well. However such policies are limited to small-scale horizons that are often discretized with extended granularity resolution [9], [27].
Besides this, other notable approximation techniques [28]- [31] also involve Bayesian optimization of cost functions by formulating a conventional transform function type approach towards specific gradient trajectories with their covariance already estimated. Such approaches convert sparsity of rewards into probability map given that rewards are of finite and discounted nature. Next step is to take account of calculated covariance and apply traversal algorithm and update policy in that direction. The algorithm takes current and predicted probabilities and calculates the Bayesian optimality recursively. When it comes to integrating more "moving variables" for decision making and taking actions in continuous horizon, recent works intent to use deeper and larger systems that can simultaneously perform end-to-end control for wide range of task in parallel [29], [32], [33].
Such methods normally are build over networked systems like convolutional neural networks (CNN) with extended array of parameters that are trained using a guided policy search method [34] that represent the policy learning as supervised approach guided by the trajectory [32], [35]. Such methods are also seen to solve real-world problems like putting a cap on bottle [32] or stacking Lego blocks [36] Talking about hierarchies, researchers also consider RL problems as modular task and claim that every problem is actually composed of concurrent sub-problems with a matter of abstraction levels [37]. Modular RL or MRL treats agents with set of sub-goals that can possibly conflict with other agent's goal as well. In such setting every sub-agent reports unique numerical preference to a meta-controller and then meta-controller assigns an action as whole to agent systems. One of the foundation study for multi-stage learning was presented as options framework by Rich. S. Sutton [18] which can be seen as a two stage system that learns abstract actions by selecting options over temporal sequences. An upgrade to options is presented as policy sketches [38] that annotate global goals as sequential sub-global task providing guidance at high level [39], [40]. Policy sketch method represent every sub-goal with a dedicated learned sub-policy that jointly maximizes a shared policy parameter among all policies; maximizing expected reward for global policy tied to shared parameter as well. Policy optimization for these sketches is done by actor-critic gradient computation [41].
We constitute the learning problem into layers by representing the a single complex agent into smaller and identical multi-agent models. We train a sub-agent independently within its own horizon, and then combine all sub-agents together by constructing a global policy based on composition of local policies. Each sub-agent is decomposed heuristically in such a way that it reminiscent a scaled-down physical structure of subsequent local agents connected in the global system-this representation is shown in fig 2 . For our test case we implement our idea on a serpentine robot with 12-DOF joints to mimic octopus like maneuverability with control over redundant joints. Our goal is to control servo maneuvers in dynamic environment, which is a complex objective to be learned as a unified problem due to hyperredundant servo joints. The redundancy is necessary to achieve bio-inspired gait but it comes with a problem of curse-of-dimensionality which exponentially increases the action space for an agent. We find serpentine design interesting because it can be seen as a chain of ideally connected small joint motors like a bio-inspired snake or octopus and the complexity can be understood by assuming taking a single joint action changes the decisions of future action selection entirely for every motor joint in the robot. Since each local agent represents a physical link in the robotic arm, they cannot move independently from each other. In other words, they are subjected to geometric constraints. This representation differentiates greatly from virtual agent-based multi-agent systems, such as two-player chess.
We divide the learning architecture into local and global levels. The local level considers an intercepted version of serpentine model (6-DOF joints in our case as shown with different colors in fig 2). At local level, the sub-agent learns a local policy π local for independent maneuver. The policy π local can be considered as a generic local control policy that can be used for any local agent in the system since each agent share similar physical structure. However local agents are constrained with movement of other sub-agents so the local policy cannot be reused with all sub-agents. Instead of using the local policy barely with agents we can parameterize it over independent policy parameters for each local agent. This parameterization is done on global level such that global policy π * learns a parameter vector -→ θ sampled over distribution of local policy which was learned by a single sub-agent and now being used by all local agents but with different global parameters. The action space for global policy is the sub-goals combination for connected local agents. Global action space can be defined as a vector G =⇒ (ĝ 1 . . . ĝn ) ∈ G.
. . .
The global policy chooses sub-goals from G and then parameterize those sub-goals with X =⇒ (θ 1 . . . θ n ∈ (0, 1). X is the parameter vector for global action. Action space is then given as
A complete global action (ĝ ∈ G, θ ∈ X ) is build with sampling a parameter θ from π p (X |S) and using parameters over sub-goal vector π p (X |S). The global policy constructs the sub-goal trajectory that lead the final end-effector to terminal state or the global goal. Globally, parameter learning and action-selection (sub-goals) is learnt by alternate updating between parameter policy and action-value function iteration. We can formulate the global learning as an MDP S, G, R, γ where reward R is a function of intrinsic signals from subagents that correspond to success at intercept level -however that does not assures success achieved globally in general.
The intrinsic reward inherently motivates the global gradient to yield better sub-goals by signaling binary value that is expressed within global reward function. In order to maintain global success, the reward function contains a predictate coefficient that is multiplied with the overall intrinsic reward.
The predictate helps exploration to logically transition towards the sub-goal selection that is not just optimal intrinsically but returns optimality towards global goal as well. This approach shadows the surrogate technique of proximal policy optimization or PPO [42]. But PPO is constrained with continuous model interaction during training which is expensive for dexterous models like snake robots. Our constructive method assumes that the local policy is well-trained over intercepted sub-agent and only requires a parametric approximation to be used over subsequent connected subagents. The global layer works as a moderator 3 to monitor the sug-goals and adapt the gradients for optimal parameters over their probability distributions. ∵ P (a i ) = a i log(θ i ) + (1 -a i ) log(1 -θ i ) For simplicity, assume all agents choose parameter θ i then for single agent i update:
Partial derivatives with respect to the parameter θ for value functions in and logarithmic transition probability gives simplified expression for value function that can be iterated recursively in place of classic bellman function reducing effort by utilizing localized probability distributions over shared θ parameters
We carry out and validate our experiments over a hyperredundant serpentine robot arm with 12-DOF maneuvering capabilities. The arm resembles maneuvering properties of an octopus' tentacles. We performed experiments against Deterministic Proximal Policy Optimization (DPPO) method and Asynchronous Advantage Actor-Critic (A3C) method. Both methods have their own benefits in different domains; A3C provides an edge to process the algorithm over very low computing power and uses parallel threads over CPU instead of memory-hungry gradient update methods. However A3C does not provides leverage of experience replay buffers which relinquishes it's performance in sparse rewards behaviors which can be seen in fig: 5. Also looking at DPPO success rate, it is apparent that DPPO method has a promising gradient which verifies [43] icy optimization updates alternate between sampling data through interaction with the environment, and optimizing a "surrogate" objective function using stochastic gradient ascent [42]. However for problems that involve dexterous manipulation like serpentine control the environment interaction is exhaustive and less likely to happen and this can be seen by the success rate of DPPO in Table I.
In contrast with DPPO and A3C, our proposed Constructive policy method performs relatively better and requires less time complexity to give better success rate. Quicker convergence trend stabilizes the effect of global policy learning for control of high-dimensional architecture in environments with rewards that are sparsely distributed. Our testing environment was setup within ROS ecosystem using Gazebo simulator for ROS to train and test robot models. ROS environment has proven to be standardized method to validate robot simulations [33]. We tested the performance over both 2D and 3D continuous spaces with Gazebo world. Furthermore we have also designed a physical robot arm for executing telemetries of simulation world over real world. Telemetry tests have been done, however our future work involves realtime testing of Constructive RL algorithm. Fig. 5: Comparison plot for success rate (finish at terminal state) over 5000 epochs. A3C has many steep but slower increments, and DPPO shows quick but deferred progress. In long run, our proposed constructive policy method outperforms both methods and gives a success rate >74% Fig. 6: Moving reward for proposed method vs. off-the-shelf DPPO. Though not worse but visibly DPPO has higher jitter in reward.
Our study contributes a promising approach in area of hierarchical deep reinforcement learning specially for complex robotic arms that are inspired by neurophysiology of living beings. Hyper-redundant robotic arms possess intricate designs and require exhaustive heuristics to formulate control functions. We believe that "constructive policy learning" contributes towards formal simplification of control task for bio-inspired robots by allowing structured composibility to divide high-dimensional problem spaces into significantly identical, smaller agents and treat unified control problem in distributed manner. Constructive policy method can be applied in environment where rewards are sparse or often accumulated with larger delaying factor -in contrast methods like -Greedy search or vanilla policy methods arguably deteriorate the expected reward outcome in sparse setting. Our method operates in distributed setting allowing local agent to share their states in entirety with global moderator such that the resulting global policy is fused with shared feedback of local agents' rewards -offering offline estimate or positive bias for Q-Value. We believe constructive Policy RL approach can be most useful in problems that can be decoupled into smaller sub-problems. Quick examples would be teaching a robot arm to open drawer, or teaching an octopus to slide-climb -dividing training process into "wrap around an object", then "push itself against object to climb". We have tested our idea in real-world simulation environment (ROS Gazebo) with active physics conditions and kinematics requirements. Our testing subject was a simulation robot arm that has 12 joints operating in continuous state and action space. Given any location within robot arm's workspace -it navigates to that Cartesian coordinate location. We observed success rate greater than 74% in simulation. We believe that this work along with recent advancements in connected robot learning area can contribute significantly towards greater developments. Also, our on-going work involves testing our method over a real physical robot (fig: 1) that has identical joints as simulation model.