fulyvv/pose2sim
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

JOSS test

Browse Source
This commit is contained in:
davidpagnon 2022-08-19 14:06:31 -07:00
parent fe4cfba6f7
commit de8d882002
1 changed files with 7 additions and 12 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

19
Content/paper.md
View File

@ -43,17 +43,15 @@ The repository presents a framework for: \
Each task is easily customizable, and requires only moderate Python skills. `Pose2Sim` is accessible at [https://github.com/perfanalytics/pose2sim](https://github.com/perfanalytics/pose2sim).
# Statement of need
For the last few decades, marker-based kinematics has been considered the best choice for the analysis of human movement, when regarding the trade-off between ease of use and accuracy. However, a marker-based system is hard to set up outdoors or in context, and it requires placing markers on the body, which can hinder natural movement [Colyer_2018].
For the last few decades, marker-based kinematics has been considered the best choice for the analysis of human movement, when regarding the trade-off between ease of use and accuracy. However, a marker-based system is hard to set up outdoors or in context, and it requires placing markers on the body, which can hinder natural movement [@Colyer_2018].
The emergence of markerless kinematics opens up new possibilities. Indeed, the interest in deep-learning pose estimation neural networks has been growing fast since 2015 [@Zheng_2022], which makes it now possible to collect accurate and reliable kinematic data without the use of physical markers. OpenPose, for example, is a widespread open-source software which provides 2D joint coordinate estimations from videos. These coordinates can then be triangulated in order to produce 3D positions with a number of available tools such as the native OpenPose 3D reconstruction module [@Hidalgo_2021], the FreeMoCap Python and Blender toolbox [@Matthis_2022], or the pose3d Matlab toolbox [@Sheshadri_2020]. Yet, when it comes to the biomechanical analysis of human motion, it is often more useful to obtain joint angles than joint center positions in space. Joint angles allow for better comparison among trials and individuals, and they represent the first step for other analysis such as inverse dynamics.
The emergence of markerless kinematics opens up new possibilities. Indeed, the interest in deep-learning pose estimation neural networks has been growing fast since 2015 [@Zheng_2022], which makes it now possible to collect accurate and reliable kinematic data without the use of physical markers. `OpenPose`, for example, is a widespread open-source software which provides 2D joint coordinate estimations from videos. These coordinates can then be triangulated in order to produce 3D positions. Aside from `Pose2Sim`, a number a tools are available to do so: the experimental `OpenPose 3D reconstruction module` [@Hidalgo_2021], the `FreeMoCap` Python and Blender toolbox [@Matthis_2022], or the `pose3d` Matlab toolbox [@Sheshadri_2020]. Yet, when it comes to the biomechanical analysis of human motion, it is often more useful to obtain joint angles than joint center positions in space. Joint angles allow for better comparison among trials and individuals, and they represent the first step for other analysis such as inverse dynamics.
OpenSim is another widespread open-source software which helps compute 3D joint angles, usually from marker coordinates. It lets scientists define a detailed musculoskeletal model, scale it to individual subjects, and perform inverse kinematics with customizable biomechanical constraints. It provides other features such as net calculation of joint moments or individual muscle forces resolution, although this is out of the scope of our contribution.
`OpenSim` is another widespread open-source software which helps compute 3D joint angles, usually from marker coordinates. It lets scientists define a detailed musculoskeletal model, scale it to individual subjects, and perform inverse kinematics with customizable biomechanical constraints. It provides other features such as net calculation of joint moments or individual muscle forces resolution, although this is out of the scope of our contribution.
The goal of `Pose2Sim` is to build a bridge between the communities of computer vision and biomechanics, by providing a simple and open-source pipeline connecting the two aforementioned state-of-the-art tools: OpenPose and OpenSim.
So far, little work has been done towards obtaining 3D angles from multiple views [@Zheng_2022]. However, three software applications are worth mentioning. `Anipose` [@Karashchuk_2021] proposes a Python open-source framework which allows for joint angle estimation with spatio-temporal constraints, but it is primarily designed for animal motion analysis. `Theia3D` [@Kanko_2021] is a software application for human gait markerless kinematics. Although the GUI is more user friendly, it is not open-source nor customizable. `OpenCap` [@Ulrich_2022] has recently been released, and offers a user friendly web application working with low-cost hardware. It predicts the coordinates of 43 anatomical markers from 20 triangulated keypoints, and imports them in `OpenSim`. However, the source code has not been released yet.
`Pose2Sim` has already been used and tested in a number of situations (walking, running, cycling, balancing, swimming, boxing), and published in peer-reviewed scientific publications [@Pagnon_2021; @Pagnon_2022] assessing its robustness and accuracy. Its results on inverse kinematics were deemed good when compared to marker-based ones, with errors generally below 4.0° on both lower- and upper-limb, accross several activities. The combination of its ease of use, customizable characteristics, and high robustness and accuracy makes it promising, especially for "in-the-wild" sports movement analysis.
So far, little work has been done towards obtaining 3D angles from multiple views [@Zheng_2022]. However, three software applications are worth mentioning. Anipose [@Karashchuk_2021] proposes a Python open-source framework which allows for joint angle estimation with spatio-temporal constraints, but it is primarily designed for animal motion analysis. Theia3D [@Kanko_2021] is a software application for human gait kinematics from videos. Although the GUI is more user friendly, it is not open-source nor easily customizable. OpenCap [@Ulrich_2022] has recently been released, and offers a user friendly web application. It predicts the coordinates of 43 anatomical markers from 20 triangulated keypoints, and import them in OpenSim. However, the source code has not been released yet.
The goal of `Pose2Sim` is to build a bridge between the communities of computer vision and biomechanics, by providing a simple and open-source pipeline connecting the two aforementioned state-of-the-art tools: `OpenPose` and `OpenSim`. The whole workflow runs from any video cameras, on any computer, equipped with any operating system (although OpenSim has to be compiled from source on Linux.) `Pose2Sim` has already been used and tested in a number of situations (walking, running, cycling, balancing, swimming, boxing), and published in peer-reviewed scientific publications assessing its robustness [@Pagnon_2021] and accuracy [@Pagnon_2022]. Its results on inverse kinematics were deemed good when compared to marker-based ones, with errors generally below 4.0° across several activities, both on lower- and on upper-limb. The combination of its ease of use, customizable parameters, and high robustness and accuracy makes it promising, especially for "in-the-wild" sports movement analysis.
# Features
## Pose2Sim workflow
@ -61,19 +59,16 @@ So far, little work has been done towards obtaining 3D angles from multiple view
• OpenPose [@Cao_2019], a 2D human pose estimation neural network\
• OpenSim [@Delp_2007], a 3D biomechanics analysis software
![Pose2Sim full pipeline: (1) OpenPose 2D joint detection; (2i) Camera calibration; (2ii–iv) Tracking the person of interest, Triangulating his coordinates, and Filtering them; (3) Constraining the 3D coordinates to a physically consistent OpenSim skeletal model.\label{fig:pipeline}](Pipeline.png)
The workflow is organized as follows \autoref{fig:pipeline}: \
1. Preliminary OpenPose [@Cao_2019] 2D keypoints detection.\
2. `Pose2Sim` core, including 4 customizable steps:\
    2.i. Camera calibration. \
    2.ii. Tracking the person of interest.\
    2.ii. 2D Tracking the person of interest.\
    2.iii. 3D keypoints triangulation.\
    2.iv. 3D coordinates filtering.\
3. A full-body OpenSim [@Delp_2007] skeletal model with OpenPose keypoints is provided, as well as scaling and inverse kinematics setup files. As the position of triangulated keypoints are not dependent on either the operator nor the subject, these setup files can be used as is.
OpenPose, OpenSim, and the whole `Pose2Sim` workflow run from any video cameras, on any computer, equipped with any operating system. However, on Linux, OpenSim has to be compiled from source.
![Pose2Sim full pipeline: (1) OpenPose 2D joint detection; (2i) Camera calibration; (2ii–iv) Tracking the person of interest, Triangulating keypoints coordinates, and Filtering them; (3) Constraining the 3D coordinates to a physically consistent OpenSim skeletal model.\label{fig:pipeline}](Pipeline.png)
## Pose2Sim method details
`Pose2Sim` is meant to be as fully and as easily configurable as possible, by editing the 'User/Config.toml' file. Optional tools are also provided for extending its usage \autoref{fig:utilities}.