Pose2Sim

Please set undistort_points and handle_LR_swap to false for now since reprojection error is currently inaccurate. I'll try to fix it soon.

News: Version 0.5 released:
Deep change in the folder structure to allow for automatic batch processing!
The Stanford LSTM marker augmentation is now supported! (Thank you @rlagnsals)

To upgrade, type pip install pose2sim --upgrade.

Pose2Sim provides a workflow for 3D markerless kinematics, as an alternative to the more usual marker-based motion capture methods. It aims to provide a free tool to obtain research-grade results from consumer-grade equipment. Any combination of phone, webcam, GoPro, etc can be used.

Pose2Sim stands for "OpenPose to OpenSim", as it uses OpenPose inputs (2D keypoints coordinates obtained from multiple videos) and leads to an OpenSim result (full-body 3D joint angles). Other 2D pose estimators such as BlazePose, DeeplLabCut, AlphaPose, etc. can now be used as inputs.

If you can only use one single camera and don't mind losing some accuracy, please consider using Sports2D.

N.B.: As always, I am more than happy to welcome contributors (see How to contribute).

Pose2Sim releases:

• v0.1: Published online
• v0.2: Published associated paper
• v0.3: Supported other pose estimation algorithms
• v0.4: New calibration tool based on scene measurements
• v0.5: Automatic batch processing
• v0.6: Marker augmentation, Handling left/right swaps, Correcting lens distortions
• v0.7: Calibration based on keypoint detection
• v0.8: Supports multi-person analysis
• v0.9: New synchronization tool
• v0.10: Graphical User Interface
• v1.0: First accomplished release

Contents

1. Installation and Demonstration
   1. Installation
   2. Demonstration Part-1: Build 3D TRC file on Python
   3. Demonstration Part-2: Obtain 3D joint angles with OpenSim
   4. Demonstration Part-3 (optional): Visualize your results with Blender
2. Use on your own data
   1. Setting your project up
      1. Retrieve the folder structure
      2. Batch processing
   2. Camera calibration
      1. Convert from Qualisys, Optitrack, Vicon, OpenCap, EasyMocap, or bioCV
      2. Calculate from scratch
   3. 2D pose estimation
      1. With OpenPose
      2. With Mediapipe
      3. With DeepLabCut
      4. With AlphaPose
   4. Camera synchronization
   5. Tracking, Triangulating, Filtering
      1. Associate persons across cameras
      2. Triangulating keypoints
      3. Filtering 3D coordinates
      4. Marker augmentation
   6. OpenSim kinematics
      1. OpenSim Scaling
      2. OpenSim Inverse kinematics
      3. Command Line
3. Utilities
4. How to cite and how to contribute
   1. How to cite
   2. How to contribute and to-do list

Installation and Demonstration

Installation

1. Install OpenPose (instructions there).
   Windows portable demo is enough.

2. Install OpenSim 4.x (there).
   Tested up to v4.4-beta on Windows. Has to be compiled from source on Linux (see there).

3. Optional. Install Anaconda or Miniconda.
   Open an Anaconda terminal and create a virtual environment with typing:

   conda create -n Pose2Sim python=3.8 -y 
   conda activate Pose2Sim

4. Install Pose2Sim:
   If you don't use Anaconda, type python -V in terminal to make sure python>=3.8 is installed.

   • OPTION 1: Quick install: Open a terminal.

     pip install pose2sim
     

   • OPTION 2: Build from source and test the last changes: Open a terminal in the directory of your choice and Clone the Pose2Sim repository.

     git clone --depth 1 https://github.com/perfanalytics/pose2sim.git
     cd pose2sim
     pip install .
     

Demonstration Part-1: Build 3D TRC file on Python

This demonstration provides an example experiment of a person balancing on a beam, filmed with 4 calibrated cameras processed with OpenPose.

Open a terminal, enter pip show pose2sim, report package location.
Copy this path and go to the Demo folder with cd <path>\pose2sim\Demo\S00_Demo_Session.
Type ipython, and test the following code:

from Pose2Sim import Pose2Sim
Pose2Sim.calibration()
Pose2Sim.personAssociation()
Pose2Sim.triangulation()
Pose2Sim.filtering()
Pose2Sim.markerAugmentation()

3D results are stored as .trc files in each trial folder in the pose-3d directory.

N.B.: Default parameters have been provided in Config.toml but can be edited.
N.B.: Try calibration tool by changing calibration_type to calculate instead of convert (more info there).

Demonstration Part-2: Obtain 3D joint angles with OpenSim

In the same vein as you would do with marker-based kinematics, start with scaling your model, and then perform inverse kinematics.

Scaling

1. Open OpenSim.
2. Open the provided Model_Pose2Sim_LSTM.osim model from Pose2Sim/OpenSim_Setup. (File -> Open Model)
3. Load the provided Scaling_Setup_Pose2Sim_LSTM.xml scaling file from Pose2Sim/OpenSim_Setup. (Tools -> Scale model -> Load)
4. Run. You should see your skeletal model take the static pose.

Inverse kinematics

1. Load the provided IK_Setup_Pose2Sim_LSTM.xml scaling file from Pose2Sim/OpenSim_Setup. (Tools -> Inverse kinematics -> Load)
2. Run. You should see your skeletal model move in the Vizualizer window.
   

Demonstration Part-3: Visualize your results with Blender

Visualize your results and look in detail for potential areas of improvement (and more).

Install the add-on

Follow instructions on the Pose2Sim_Blender add-on page.

Visualize your results

Just play with the buttons!

N.B.: You need to proceed to the full install to import the inverse kinematic results from OpenSim. See instructions there.