That is interesting.
If the goal is to “control a humanoid robot”. it makes perfect sense and I can’t think of anything better (apart from a bilateral BCI)
But the concept here is to “serve as a ‘universal’ controler for multiple distinct devices”… Then let’s ignore the “predictive” aspect for now.
So… let’s assume BCI’s would be the pinnacle of “intuitive” control.
Therefore, if we’re deliberately dodging a BCI and going for the “analog” counterpart (haptic feedback, audio, VR headsets), then we do have a few possible options.
One of which is a “suit” which uses bodily movements and more (buttons on the wrists, etc).
Another would be a cockpit…
Another, a home Desktop computer… etc, etc…
In order to better analyze what were getting at, let’s split this device into input (what you do that generates an input into the controlled device) and feedback (the audio/video/haptic feedback you receive).
A VR headset is nice, but having used extensively both VR and Desktops (my desktop has between 4 to 6 monitors)… I’ll tell you that VR is by far more limited than an array of monitors.
This is due to how the VR headsets currently in existence work, which causes an illusion of immersion, but the actual field of view is narrowed a lot and you can only read comfortably what is placed right in front of you.
Hence an array of monitors would provide you with far more information within your natural view than anything within VR would.
You would also face a Depth of field issue. In VR you have problems adjusting your view automatically to heads up displays which are mainly separated by depth. this occurs because of the way the lenses are assembled on the headset.
With a cockpit or array of monitors you can overlay information in a much more natural and intuitive manner.
Add some Eye tracking and servo motors on your monitor arms and we could even have a cool system where the monitors come and go from your field of view based on what you’re looking at.
That could provide a much more natural and intuitive feedback for the majority of devices you would want to control…
Binaural audio is brilliant and all you need for that is a headset.
Not much more to say here.
I think I get your premise for “extending your perception and being able to feel like what you are controlling”.
Let’s keep up on the “piloting a spacecraft” example:
Even with a Bilateral BCI it’s still only in theory that you could “feel the wind rushing through your wings”…
With a haptic suit at most you could get to “feel some pressure on your arm, which correlates to the pressure of the wind on the aircraft”.
And even then, you would still need a special aircraft with sensors on the wing, and which could convey such data back to the controller… not “any device” anymore.
Let’s take another example: driving an armored car. The car gets shot multiple times on the right rear door…
How should that translate as information on a haptic suit?
I mean… what would be “intuitive” about any haptic feedback there?
The way I see it, a haptic suit would be mostly a constant reminder of our human form, instead of means to transcend it, because as soon as you let yourself get immersed into piloting something nonhuman… if there is haptic feedback it would just make you painfully aware of your physicality.
The best and most intuitive input for any device is usually custom tailored for said device.
Take a plane drone for example: You can technically control it using a keyboard but that would be far clunkier than to control it using a Handle…
On the other hand, try to write a message for someone with a handle?
But again, we are talking about a universal controller, so the best we have at the moment is a computer’s keyboard.
If we want to push it forward, imagine a setup with a full keyboard, 2 dedicated smaller keysets, a mouse, a joystic, and toss in some motion tracking for gesture commands, and touchscreens on your monitors…
Now let’s compare that to what we could put into a suit.
We can work buttons/servos activated by moving the fingers…
That is perfect for controlling a humanoid hand, or to trigger a few simple commands…
But you could achieve the same with motion capture, without the need of a clunky set of gloves.
Even more, such a set of controllers would serve as a similar interface to a keyboard, only much more limited.
With the suit we could also use full body movements… But at the same time using large movements is limiting. i.e. if you need to stretch an arm forward to accelerate a car, then you can’t use that hand for much else…
Imagine trying to use your legs to control something… you would end up walking around the house and hit the wall… XD
Then let’s bring “Intuitive” to the mix…
What’s more intuitive when controlling a car?
- pressing forward on a keyboard?
- touching your pinky with your thumb while pointing up with the index finger?
- stretching your arm forward?
With a bilateral BCI we could have a translation which, in theory, would make us control the car directly in a purely intuitive manner.
But with haptics and servos, all you can capture are movements and gestures… which you can also capture with motion tracking. But that doesn’t make controlling an inhuman shape any particularly intuitive endeavour…
That’s where I see the split on your project as a focal point you would need to make a decision for:
Is this meant to be something to control a humanoid shape? or is this meant to be a universal control module to allow the user to “plug in” into other devices (cars, spacecrafts, etc…) ??
Even further… is this meant to be something to help you control other devices as an extension of yourself (which, arguably, is what we use our phones for)?
And if so, is the focus on allowing you to control said devices better than anything else would, or is the purpose to make you feel like you’re being something else, even if you’re not controlling something?
Because for the later, then a VR videogame with a good psychologically backed design would achieve this far better than any “universal” or modular device could… (exactly because part of what causes the immersion effect while playing VR is the fantastic setting in which you are and the lack of physical feedback. VR feedback must be either perfectly precise, or absent so that we can let our imagination run free)