{ "cells": [ { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 4. Computing FLOPS, latency and fps of a model\n\nIt is important to have an idea of how to measure a video model's speed, so that you can choose the model that suits best for your use case.\nIn this tutorial, we provide two simple scripts to help you compute (1) FLOPS, (2) number of parameters, (3) fps and (4) latency.\nThese four numbers will help you evaluate the speed of this model.\nTo be specific, FLOPS means floating point operations per second, and fps means frame per second.\nIn terms of comparison, (1) FLOPS, the lower the better,\n(2) number of parameters, the lower the better,\n(3) fps, the higher the better,\n(4) latency, the lower the better.\n\n\nIn terms of input, we use the setting in each model's training config.\nFor example, I3D models will use 32 frames with stride 2 in crop size 224, but R2+1D models will use 16 frames with stride 2 in crop size 112.\nThis will make sure that the speed performance here correlates well with the reported accuracy number.\nWe list these four numbers and the models' accuracy on Kinetics400 dataset in the table below.\n\n\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| Model | FLOPS | # params | fps | Latency | Top-1 Accuracy |\n+========================================+================+============+=====================+=================+=================+\n| resnet18_v1b_kinetics400 | 1.819 | 11.382 | 264.01 | 0.0038 | 66.73 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| resnet34_v1b_kinetics400 | 3.671 | 21.49 | 151.96 | 0.0066 | 69.85 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| resnet50_v1b_kinetics400 | 4.110 | 24.328 | 114.05 | 0.0088 | 70.88 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| resnet101_v1b_kinetics400 | 7.833 | 43.320 | 59.56 | 0.0167 | 72.25 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| resnet152_v1b_kinetics400 | 11.558 | 58.963 | 36.93 | 0.0271 | 72.45 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| i3d_resnet50_v1_kinetics400 | 33.275 | 28.863 | 1719.50 | 0.0372 | 74.87 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| i3d_resnet101_v1_kinetics400 | 51.864 | 52.574 | 1137.74 | 0.0563 | 75.10 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| i3d_nl5_resnet50_v1_kinetics400 | 47.737 | 38.069 | 1403.16 | 0.0456 | 75.17 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| i3d_nl10_resnet50_v1_kinetics400 | 62.199 | 42.275 | 1200.69 | 0.0533 | 75.93 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| i3d_nl5_resnet101_v1_kinetics400 | 66.326 | 61.780 | 999.94 | 0.0640 | 75.81 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| i3d_nl10_resnet101_v1_kinetics400 | 80.788 | 70.985 | 890.33 | 0.0719 | 75.93 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| i3d_slow_resnet50_f8s8_kinetics400 | 41.919 | 32.454 | 1702.60 | 0.0376 | 74.41 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| i3d_slow_resnet50_f16s4_kinetics400 | 83.838 | 32.454 | 1406.00 | 0.0455 | 76.36 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| i3d_slow_resnet50_f32s2_kinetics400 | 167.675 | 32.454 | 860.74 | 0.0744 | 77.89 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| i3d_slow_resnet101_f8s8_kinetics400 | 85.675 | 60.359 | 1114.22 | 0.0574 | 76.15 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| i3d_slow_resnet101_f16s4_kinetics400 | 171.348 | 60.359 | 876.20 | 0.0730 | 77.11 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| i3d_slow_resnet101_f32s2_kinetics400 | 342.696 | 60.359 | 541.16 | 0.1183 | 78.57 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| r2plus1d_v1_resnet18_kinetics400 | 40.645 | 31.505 | 804.31 | 0.0398 | 71.72 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| r2plus1d_v1_resnet34_kinetics400 | 75.400 | 61.832 | 503.17 | 0.0636 | 72.63 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| r2plus1d_v1_resnet50_kinetics400 | 65.543 | 53.950 | 667.06 | 0.0480 | 74.92 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| r2plus1d_v2_resnet152_kinetics400 | 252.900 | 118.227 | 546.19 | 0.1172 | 81.34 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| ircsn_v2_resnet152_f32s2_kinetics400 | 74.758 | 29.704 | 435.77 | 0.1469 | 83.18 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| slowfast_4x16_resnet50_kinetics400 | 27.820 | 34.480 | 1396.45 | 0.0458 | 75.25 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| slowfast_8x8_resnet50_kinetics400 | 50.583 | 34.566 | 1297.24 | 0.0493 | 76.66 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| slowfast_8x8_resnet101_kinetics400 | 96.794 | 62.827 | 889.62 | 0.0719 | 76.95 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| tpn_resnet50_f8s8_kinetics400 | 50.457 | 71.800 | 1350.39 | 0.0474 | 77.04 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| tpn_resnet50_f16s4_kinetics400 | 99.929 | 71.800 | 1128.39 | 0.0567 | 77.33 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| tpn_resnet50_f32s2_kinetics400 | 198.874 | 71.800 | 716.89 | 0.0893 | 78.90 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| tpn_resnet101_f8s8_kinetics400 | 94.366 | 99.705 | 942.61 | 0.0679 | 78.10 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| tpn_resnet101_f16s4_kinetics400 | 187.594 | 99.705 | 754.00 | 0.0849 | 79.39 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n| tpn_resnet101_f32s2_kinetics400 | 374.048 | 99.705 | 479.77 | 0.1334 | 79.70 |\n+----------------------------------------+----------------+------------+---------------------+-----------------+-----------------+\n\n\n
Feel free to skip the tutorial because the speed computation scripts are self-complete and ready to launch.\n\n :download:`Download Full Python Script: get_flops.py<../../../scripts/action-recognition/get_flops.py>`\n\n :download:`Download Full Python Script: get_fps.py<../../../scripts/action-recognition/get_fps.py>`\n\n You can reproduce the numbers in the above table by\n\n ``python get_flops.py --config-file CONFIG`` and ``python get_fps.py --config-file CONFIG``\n\n If you encouter missing dependecy issue of ``thop``, please install the package first.\n\n ``pip install thop``