Whenever you are doing an activity where you touch hands with another person and move in an interactive fashion, it helps to remember that the point where you are touching each other is NOT the important part. When you touch the other person, you need to feel their entire body down to the ground in order to truly affect their movement and interact with them.
You should also be moving from your feet. Your power comes from the ground. If you move from the arms or the top of the torso by scrunching up the body, your ability to control the interaction is limited. If you don't feel them down to their feet, you won't be affecting their whole body and balance.
If that doesn't make sense, think of it this way...Put your hands out and have your partner put his or her hands out. Move forward and don't change the arm position until you can feel the persons feet (that means you could make them move their feet with a small amount of pressure). When you can find their feet, make them move by using your feet pushing against the ground. Only then should you move your arms in coordination with the feet.
If this still doesn't make sense, just focus on this: we connect up with someone to interact or fight through touching. The point we touch is a way to feel deeper into the person. Don't get stuck just trying to move the connection point. The cable between your television and the electric outlet isn't important. The socket and the tv are.