OpenCV 3D module rendering function PR by nickyu-zhu · Pull Request #24065 · opencv/opencv

nickyu-zhu · 2023-07-26T18:02:01Z

Pull Request for GSoC 2023 Project5 - Simple Triangle Rendering

This is a pull request for OpenCV 3d module in 5.x branch, which is the work of GSoC 2023 project 5. The changes are demonstrated down below.

Create rendering.cpp and rendering.hpp files in opencv/modules/src folder, which can render the input triangles through the triangleRastrize() function. This new function can now perform depth and color rendering successfully
Create test_rendering.cpp file in the opencv/modules/test folder, which test the triangleRastrize() function in 4 different ways, (including empty set, constant color rendering, interpolated color rendering and clipping test rendering)
Add three new files in opencv/sample/opengl folder. The names are opengl_clipping_sample.cpp, opengl_color_sample.cpp and opengl_depth._sample.cpp. These files are set to be the OpenGL ground truth samples for the rendering image.

* bmp specified BI_BITFIELDS should take care RGBA bit mask * support xrgb bmp file

…ale image

add test case for "tf/reduce_sum" * fix output dimension of reduce_sum * add test case and fix bug * add more cases * style:change files name * fix a little bug

Add regression test for QRcode encoding and decoding Co-authored-by: APrigarina <ann73617@gmail.com>

Signed-off-by: Crayon-new <1349159541@qq.com>

Onnx conformance tests * Add ONNX conformance tests * dnn(onnx): make conformance stubs a part of repository Co-authored-by: Alexander Alekhin <alexander.a.alekhin@gmail.com>

Signed-off-by: Crayon-new <1349159541@qq.com>

…_ratio

savuor

This is a good progress. There's a couple of things to fix:

My comments
Broken CI tests

savuor · 2023-08-27T20:50:56Z

 */
 CV_EXPORTS_W void saveMesh(const String &filename, InputArray vertices, InputArray normals, InputArrayOfArrays indices);

+/*


This comment does not render to docs.
You can always check the generated documentation for the PR here (Builders -> default -> PR number -> Docs -> Upload docs -> Preview).

savuor · 2023-08-27T23:19:08Z

@@ -0,0 +1 @@
+


We don't need empty files, please remove

savuor · 2023-08-28T00:48:08Z

+    {
+        if (debugLevel > 0)
+        {
+            Mat groundTruth = imread("../../../opencv_extra/opencv_extra/testdata/rendering/example_image_depth_1.png");


This file path will work on your machine only.
We have a special function to obtain a path:

// The file path is OPENCV_TEST_DATA_PATH + "/rendering/example_image_depth_1.png" std::string dataPath = cvtest::TS::ptr()->get_data_path() + "/rendering/example_image_depth_1.png";

The test executables should be run with env variable OPENCV_TEST_DATA_PATH set to testdata folder

findDataFile() should be used.

savuor · 2023-08-28T01:08:04Z

+
+        std::vector<Vec3f> verticesVector = vertices.getMat();
+        std::vector<Vec3f> indicesVector = indices.getMat();
+        std::vector<Vec3f> colorsVector = colors.getMat();


We should allow an empty vertex colors array. In that case all the triangles will be drawn by the same color (let it be full white).

savuor · 2023-08-28T01:28:27Z

+* */
+CV_EXPORTS  void triangleRasterize(InputArray vertices, InputArray indices,
+    InputArray colors, InputArray cameraMatrix, int width, int height, bool shadingMode,
+    OutputArray depth_buf, OutputArray color_buf);


OK, the thing happened to cameraMatrix is not what I was talking about. It should be changed:

The way of passing camera pose by lookAt, position and up vector is not an OpenCV-style, we use camera pose matrix for that (in your code this is called lookAtMatrix

The intrinsic camera parameters are also passed in different way, by 3x3 intrinsic matrix
I'll rewrite the doc string, and you are to change the function signature according to it.

Renders a set of triangles on a depth and/or RGB image. The output images are not cleared before the rendering and can be used for masking or with pre-filled Z-buffer. Triangles can be flat-shaded or have interpolated colors between vertices. In flat-shaded mode a color of 1st triangle vertex is used. If no colors are given, the triangles are drawn in white (1.0, 1.0, 1.0). @param vertices vertices coordinates array. Should contain values of CV_32FC3 type or a compatible one (e.g. cv::Vec3f, etc.) @param indices triangle vertices index array, 3 per triangle. Each index indicates a vertex in a vertices array. Should contain CV_32SC3 values @param colors per-vertex colors of CV_32FC3 type, can be empty. @param cameraPose a 4x3 or 4x4 float matrix containing camera pose @param fovY field of view in vertical direction, given in degrees @param zNear minimum Z value to render, everything closer is clipped @param zFar maximum Z value to render, everything farther is clipped @param width Frame width @param height Frame height @param useFlatShading Enables or disables flat shading *@param depthBuf a width x height array of floats containing resulting Z buffer. Reused if not empty. Created and filled by zFar values if a user-provided array is empty. To disable Z buffer output, pass cv::noArray() here. *@param colorBuf a width x height array of CV_32FC3 representing the final rendered image. Reused if not empty. Created and filled by zeroes if a user-provided array is empty. To disable color output, pass cv::noArray() here.

CV_EXPORTS void triangleRasterize(InputArray vertices, InputArray indices, InputArray colors, InputArray cameraPose, float fovY, float zNear, float zFar, int width, int height, bool useFlatShading, OutputArray depthBuf=noArray(), OutputArray colorBuf=noArray());

savuor · 2023-08-29T01:34:04Z

+        Vec3f position = camera.row(0);
+        Vec3f lookat = camera.row(1);
+        Vec3f upVector = camera.row(2);
+        float fovy = camera.at<float>(3, 0), zNear = camera.at<float>(3, 1), zFar = camera.at<float>(3, 2);


Please change this according to new function signature

savuor · 2023-08-29T01:38:06Z

+        Matx44f mvpMatrix = perspectMatrix * lookAtMatrix;
+
+        Mat depth_buf_mat = depth_buf.getMat();
+        Mat color_buf_mat = color_buf.getMat();


Both depthBuf and colorBuf OutputArray arguments should be checked for size, type, empty() and needed()

If they are empty, they should be created with given width and height

If color buffer is not needed then the color rendering should be disabled

If depth buffer is not needed then we still need it during rendering, just don't pass it outside

savuor · 2023-08-29T01:56:27Z

+        return Vec3f(a[0] / length, a[1] / length, a[2] / length);
+    }
+
+    Matx44f lookAtMatrixCal(const Vec3f& position, const Vec3f& lookat, const Vec3f& upVector)


As I said earlier, this function makes more sense in test/perf files rather than here.
As an option, we can expose it to public interface

savuor · 2023-08-29T02:03:22Z

+        if (ACcrossAB.dot(ACcrossAP) >= 0 && ABcrossAC.dot(ABcrossAP) >= 0)
+        {
+            float beta = norm(ACcrossAP) / norm(ACcrossAB);
+            float gamma = norm(ABcrossAP) / norm(ABcrossAC);


norm(ACcrossAB) == norm(ABcrossAC), this can be optimized

If the triangle is degenerate or close to, we'll get division by zero problem, this should be avoided

savuor · 2023-08-29T02:05:24Z

+        if (ACcrossAB.dot(ACcrossAP) >= 0 && ABcrossAC.dot(ABcrossAP) >= 0)
+        {
+            float beta = norm(ACcrossAP) / norm(ACcrossAB);
+            float gamma = norm(ABcrossAP) / norm(ABcrossAC);


The same about div by zero

VIT track(gsoc realtime object tracking model) opencv#1088 add an onnx model for opencv vttrack PR opencv#24201

* add converted conformance gemm tests * update to modern googlenet with new outputs * correct filename * replace model link with the one from org gdrive * drop googlenet 2023

opencv-alalek · 2023-09-22T06:50:17Z

-    #include <windows.h>
+#define WIN32_LEAN_AND_MEAN 1
+#define NOMINMAX 1
+#include <windows.h>


avoid unnecessary changes

opencv-alalek · 2023-09-22T06:51:00Z

@@ -0,0 +1,212 @@
+#include "precomp.hpp"


Missing license header

opencv-alalek · 2023-09-22T06:51:25Z

+namespace cv {
+
+    struct Triangle
+    {
+        Vec4f vertices[3];
+        Vec3f color[3];


avoid indentation in namespaces (all files)

opencv-alalek · 2023-09-22T06:53:30Z

+    {
+        if (debugLevel > 0)
+        {
+            Mat groundTruth = imread("../../../opencv_extra/opencv_extra/testdata/rendering/example_image_depth_1.png");


findDataFile() should be used.

opencv-alalek · 2023-09-22T06:54:27Z

+struct DrawData
+{
+	ogl::Arrays arr;
+	ogl::Buffer indices;


Don't use tabs in source code.
Follow OpenCV coding style guidelines.

* add expand conformance tests * fix clip-vit-base-head model and its output

Model and test data for the fix of shared const inputs

dnn: download_models script improvements opencv#1084 Added new features: * Use argument parser and new arguments: * `--cache` directory will be tried before downloading from internet * `--dst` change destination directory * `--list` list all model names * `--cleanup` remove all `tar.gz` files afterwards * Add hierarchy to model structure, now there are regular models which can be downloaded directly and archived models which should be downloaded and then required files will be extracted. Archived files will be checked for existence before downloading whole archive - it allows to populate them from cache and avoid storing archives on disk. * Filters are now case-insensitive and use _contains_ rule for matching, multiple filters can be used in the command line, results will be united. * GDrive object simplified to separate method - allows to reduce repeated download code. Better solution would be to extract all download-store-verify functionality to separate class(es), but I decided to postpone this step for the future improvements. * Use `pathlib.Path` for most paths. I have doubts regarding filter arguments: should we keep them positional arguments or change to keyword arguments (e.g. `--filter`)? Examples: ```.sh # Populate cache and remove archives afterwards (~4Gb) ./download_models.py --dst /data/dnn --cleanup # Download models to the specified folder using cache ./download_models.py --cache /data/dnn --dst $HOME/opencv_extra/testdata/dnn # List all models having face in their name ./download_models.py --list face ```

Extend performance test models

Updated performance test sanity data for PR opencv/opencv:24333.

Add QRCode with byte data and umlaut

Merge with opencv#24386 Also noted that random seed is added in this pr, so previous data is regenerated to keep consistency.

halide cleanup

…-dnn Removed torch7 models

savuor · 2023-10-27T01:08:49Z

Reopened as #24459

Triangle rasterization function #24459 #24065 reopened since the previous one was automatically closed after rebase Connected PR with ground truth data: [#1113@extra](opencv/opencv_extra#1113) ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake

Harvey and others added 30 commits October 21, 2021 18:13

32bit rgb bmp file should not copy data as rgba

005655e

Merge pull request opencv#927 from Harvey-Huang:32bit_rgb

1079dca

Merge pull request opencv#925 from Harvey-Huang:master

9313c1b

* bmp specified BI_BITFIELDS should take care RGBA bit mask * support xrgb bmp file

Merge remote-tracking branch 'upstream/3.4' into merge-3.4

b4b84b2

fixed bug: opencv read tif file convert Palette color image as graysc…

14b350b

…ale image

Merge pull request opencv#931 from Harvey-Huang:master

e2dd637

add test data for QR detect fix

eb99255

Merge pull request opencv#933 from APrigarina:add_test_qrdetect_fix

db89432

Merge remote-tracking branch 'upstream/3.4' into merge-3.4

2f4ae84

Merge pull request opencv#932 from Crayon-new:tf_pooling

cbc4d1b

add test case for "tf/reduce_sum" * fix output dimension of reduce_sum * add test case and fix bug * add more cases * style:change files name * fix a little bug

Merge remote-tracking branch 'upstream/3.4' into merge-3.4

6a712be

Merge pull request opencv#807 from ZhengQiushi:my_3.4

127e646

Add regression test for QRcode encoding and decoding Co-authored-by: APrigarina <ann73617@gmail.com>

Merge remote-tracking branch 'upstream/3.4' into merge-3.4

0122c2f

add test data for amtmul and batchmatmul

cb19926

Signed-off-by: Crayon-new <1349159541@qq.com>

add test data for ArgMin and ArgMax

e302311

Merge pull request opencv#941 from rogday:argminmax_dnn

934397b

Merge pull request opencv#940 from pccvlab:MatMul_with_two_inputs

c5e6718

add testdata for square

3087438

Test data for new pre-filtering in cuda::StereoBM.

1171cbc

Merge pull request opencv#944 from asmorkalov:as/stereo_prefilter

d72416e

Merge pull request opencv#937 from rogday:onnx_tests

01e8a30

Onnx conformance tests * Add ONNX conformance tests * dnn(onnx): make conformance stubs a part of repository Co-authored-by: Alexander Alekhin <alexander.a.alekhin@gmail.com>

Merge pull request opencv#943 from pccvlab:square

e235126

Merge remote-tracking branch 'upstream/3.4' into merge-3.4

4b21e9d

add test data for tf_Argmin and tf_Argmax

267aafe

Signed-off-by: Crayon-new <1349159541@qq.com>

Merge pull request opencv#946 from pccvlab:tf_Arg

1304891

Regression test data for cuda::stereoBM with uniqueness check.

b1ce373

Merge pull request opencv#924 from asmorkalov:as/stereo_bm_uniqueness…

5002d3d

…_ratio

add test data for Div

b543e77

Merge pull request opencv#947 from rogday:div_fix

b2e7c8a

Merge remote-tracking branch 'upstream/3.4' into merge-3.4

5a8a68e

savuor requested changes Aug 29, 2023

View reviewed changes

fengyuentau and others added 4 commits September 6, 2023 17:03

first commit

fb831c6

Merge branch 4.x

f48d76b

Merge pull request opencv#1088 from lpylpy0514:4.x

93cdc72

VIT track(gsoc realtime object tracking model) opencv#1088 add an onnx model for opencv vttrack PR opencv#24201

add converted conformance gemm tests (opencv#1073)

c1be747

* add converted conformance gemm tests * update to modern googlenet with new outputs * correct filename * replace model link with the one from org gdrive * drop googlenet 2023

opencv-alalek reviewed Sep 22, 2023

View reviewed changes

Abdurrahheem and others added 18 commits September 22, 2023 13:05

Merge pull request opencv#1090 from Abdurrahheem:ash/dev_einsum

1aadd31

add expand conformance tests (opencv#1098)

6608980

* add expand conformance tests * fix clip-vit-base-head model and its output

add model generation with onnxscript; add model and test data

1b032da

Merge pull request opencv#1101 from fengyuentau:fix_24319

4a31f93

Model and test data for the fix of shared const inputs

Merge branch 4.x

eeae805

Add QRCode with byte data and umlaut

f2f0386

fix conflicts

8b9843f

Merge pull request opencv#1095 from WanliZhong:extend_perf_net_test

a13cf93

Extend performance test models

Merge pull request opencv#1104 from definitelyuncertain:CvtRGB2YUV422

e5892c3

Updated performance test sanity data for PR opencv/opencv:24333.

Merge pull request opencv#1105 from dkurt:py_return_non_utf8_string

00385e4

Add QRCode with byte data and umlaut

Merge pull request opencv#1107 from fengyuentau:fix_dtype_nary_eltwise

522b48c

Merge with opencv#24386 Also noted that random seed is added in this pr, so previous data is regenerated to keep consistency.

Merge pull request opencv#1092 from fengyuentau:halide_cleanup_5.x

bbc94be

halide cleanup

Merge branch 4.x

261dd8c

Removed torch7 models and files used by Torch7 tests.

0f17fd8

Merge pull request opencv#1097 from alexlyulkov:al/remove-torch7-from…

28e3fba

…-dnn Removed torch7 models

add rendering data

0a04965

fix depth map generation bug

8449258

savuor force-pushed the Ningyu-dev branch from 80add2d to 87b1a80 Compare October 27, 2023 00:29

savuor closed this Oct 27, 2023

savuor force-pushed the Ningyu-dev branch from 87b1a80 to 8449258 Compare October 27, 2023 00:42

savuor mentioned this pull request Oct 27, 2023

Triangle rasterization function #24459

Merged

6 tasks

Uh oh!

Conversation

nickyu-zhu commented Jul 26, 2023

Pull Request for GSoC 2023 Project5 - Simple Triangle Rendering

Uh oh!

savuor left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

savuor commented Oct 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

savuor commented Oct 27, 2023 •

edited

Loading