NeuroVis: Real-Time Neural Information Measurement and Visualization of Embodied Neural Systems

Left: A hexapod robot walking on a rough terrain. Right: NeuroVis displaying the neural control of the hexapod robot in real-time during walking. It displays neural spatial-temporal information of the control [i.e., neural structure (NS), neural dynamics (ND), neural plasticity (NP), and neural memory (NM)]. NeuroVis is light and can visualize neural information on a standard PC monitor and/or a mobile phone!!

If you are searching for a tool for real-time visualization of neural networks with ROS interface for XAI and robotics, you may check the open-source NeuroVis developed and recently released by our team:

Paper: https://www.frontiersin.org/articles/10.3389/fncir.2021.743101/full

Git repository: https://gitlab.com/zumoarthicha/neurovis

NeuroVis is a neural dynamic visualizer that connects to the robot operating system (ROS). It acts as a ROS node named “NeuroVis” that receives six topics as inputs: neuron name, connection, updated weight, neuron activity, normalization gain, and bias (some of them are optional); therefore, the visualizer is compatible with ROS-supported programming languages, including C++, python, and Lua. In addition, the program can be employed along with a simulation, such as CoppeliaSim (VREP), and operate in parallel to a real robot.

Abstract
Understanding the real-time dynamical mechanisms of neural systems remains a significant issue, preventing the development of efficient neural technology and user trust. This is because the mechanisms, involving various neural spatial-temporal ingredients [i.e., neural structure (NS), neural dynamics (ND), neural plasticity (NP), and neural memory (NM)], are too complex to interpret and analyze altogether. While advanced tools have been developed using explainable artificial intelligence (XAI), node-link diagram, topography map, and other visualization techniques, they still fail to monitor and visualize all of these neural ingredients online. Accordingly, we propose here for the first time “NeuroVis,” real-time neural spatial-temporal information measurement and visualization, as a method/tool to measure temporal neural activities and their propagation throughout the network. By using this neural information along with the connection strength and plasticity, NeuroVis can visualize the NS, ND, NM, and NP via i) spatial 2D position and connection, ii) temporal color gradient, iii) connection thickness, and iv) temporal luminous intensity and change of connection thickness, respectively. This study presents three use cases of NeuroVis to evaluate its performance: i) function approximation using a modular neural network with recurrent and feedforward topologies together with supervised learning, ii) robot locomotion control and learning using the same modular network with reinforcement learning, and iii) robot locomotion control and adaptation using another larger-scale adaptive modular neural network. The use cases demonstrate how NeuroVis tracks and analyzes all neural ingredients of various (embodied) neural systems in real-time under the robot operating system (ROS) framework. To this end, it will offer the opportunity to better understand embodied dynamic neural information processes, boost efficient neural technology development, and enhance user trust.

Reference: Srisuchinnawong A, Homchanthanakul J and Manoonpong P (2021) NeuroVis: Real-Time Neural Information Measurement and Visualization of Embodied Neural Systems. Front. Neural Circuits 15:743101. doi: 10.3389/fncir.2021.743101

 

Use Case I: Hexapod Robot Locomotion

Neural visualization from NeuroVis showing the overview of the NS of the adaptive neural control network (A). This neural visualization is created based on information published through ROS messages from the adaptive neural control. The ND is presented by the neurons C0, C1, DL0-DL11, CF0-CF5, and FT0-FT5 which are highlighted in blue. The NP is presented by the neurons d0-d11, which are highlighted in red. The NM is represented by the neurons S0-K0 to S5-K5, which are highlighted in green. (B) The diagram of the adaptive neural control with the highlighted neural components that are employed to present the ND, NP, and NM. (C) Simulated robot. (D) Physical robot (Thor and Manoonpong, 2019).

Neural visualization from NeuroVis showing the use of NeuroVis for robot locomotion control and reinforcement learning in a closed-loop embodied neural network. (A) Neural visualization of NeuroVis. (B) Neural control that is used in the second use case and visualized in (A). (C) Simulated and physical robots. (D) Learned optimal output signal for optimal swing and stance patterns. (E) Example of the reward signal for learning the connection weights between K1-8 and O to obtain the optimal output signal.