Multiscale mode

In order to process an image more quickly and robustly with pandora2d, it is possible to use multiscale mode. This mode launches pandora2d pipelines on a given image in different resolutions.

To use multiscale mode, you must install the 1.1.2.dev0 pandora2d version.

Multiscale mode can be launched with the following command:

python -m src.pandora2d.multiscale.multiscale data_samples/json_conf_files/a_multiscale_pipeline.json

We can use a verbose mode:

• -v option prints informations about multiscale pipeline

• -vv option prints informations about multiscale and pandora2d pipelines

Hint

It is possible to run a_multiscale_pipeline.json located in the pandora2d github repository for the 1.1.2.dev0 version.

The image pyramids used are derived from the left.tif and right.tif images in the maricopa folder that were first downsampled by a factor 4. Then these images were downsampled again by factors 4 and 2 to create pyramids of 3 images with subsampling factors of 1, 2, and 4.

To learn more about multiscale theory, please refer to this section

Warning

Currently, it is not recommended to use multiscale mode with disparity ranges that are too large or with initial disparities that vary significantly between different points.

In fact, the pandora2D cost volume is allocated according to the global minimum and maximum disparities across all points. If some points have disparity ranges that differ significantly, the cost volume will be constructed to include all the disparities to be processed for all points. This can therefore significantly increase the memory required to run pandora2D configurations.

One way to address this issue is to enable segment mode (see Segment mode).

Configuration and parameters

Parameters

Multiscale mode configuration is composed of the following keys:

Table 4 Multiscale mode configuration

Name	Description	Type	Default value	Required
multiscale	Multiscale mode parameters	dict	None	Yes
pandora2d	Paths of Pandora2d json configurations used for each resolution	str or list[str]	None	Yes

Multiscale section is composed of the following keys:

Table 5 Multiscale key properties

Name	Description	Type	Default value	Required
left[“img_pyramid”]	Path to the pyramid repository for left image	string	None	Yes
left[“mask_pyramid”]	Path to the mask pyramid repository for left image	string	None	No
right[“img_pyramid”]	Path to the pyramid repository for right image	string	None	Yes
right[“mask_pyramid”]	Path to the mask pyramid repository for right image	string	None	No
model	Model used to estimate grid deformation	dict with keys: “type” must be a str in [“pol”] “degree” must be an int	{“type”: “pol”, “degree”: 2}	No
mesh	Number of mesh to divide the image	dict with keys: “row” must be an int “col” must be an int	{“row”: 1, “col”: 1}	No
minimal_nb_pixels_per_mesh	Minimum number of pixels per mesh for it to be considered valid	int	1	No
output	Path to output directory	string	None	Yes

Note

If a single JSON file is entered in the pandora2d key, then the pandora2d pipeline corresponding to this file will be used to process all image pyramid resolutions.

However, if you enter a list of JSON files in the “pandora2d” key, then you must have as many JSON files in the list as you have images in the pyramid. In this way, each JSON file corresponds to the processing of one resolution. The first JSON file corresponds to the smallest resolution and the last to the largest.

Note

The minimal_nb_pixels_per_mesh parameter allows to set a minimum number of pixels per mesh for it to be considered valid.

When exiting multiscale mode, a MESH_validity band is added to the pandora2d validity.tif file. This band provides an indication of the mesh’s validity based on the minimum number of points specified in the minimal_nb_pixels_per_mesh parameter.

This number of pixels corresponds to the minimum number of points we wish to use to estimate our models in order to consider them sufficiently robust.

If the number of pixels used to estimate the model in a mesh is less than minimal_nb_pixels_per_mesh, then that mesh is marked as 1 (invalid) in the MESH_validity band; otherwise, it is marked as 0 (valid)

Important

The image pyramids (left and right) must be provided as input to the multiscale mode. To create image pyramids, it is possible to use tools such as gridr

Examples of configuration

Example of a multiscale pipeline configuration

{
    "multiscale":
    {
        "left" : {"img_pyramid": "../images/left_pyramid"},
        "right" : {"img_pyramid": "../images/right_pyramid"},
        "model": {"type": "pol", "degree": 2},
        "mesh": {"row":3, "col":3},
        "output":
            "a_multiscale_output"
    },
    "pandora2d":
        ["resolution_1_config.json", "resolution_2_config.json", "resolution_3_config.json"] // pandora2d pipelines for each processed resolution

}

Warning

The multiscale mode takes entire pandora2d configurations as input, but the images paths specified in the pandora2d configurations are overwritten by the images contained in the pyramids specified in the multiscale key.

Furthermore, beyond the smallest resolution, the initial disparities are replaced by the initial disparity grids estimated at the previous resolution.

Warning

Multiscale mode is not yet compatible with the use of an ROI.

Example of a multiscale pipeline configuration with masks

It is possible to add input mask in the multiscale configuration. These masks will be used as input masks for the various pandora2d configurations that will be launched.

{
    "multiscale":
    {
        "left" : {"img_pyramid": "../images/left_pyramid",
                  "mask_pyramid": "../images/left_mask_pyramid"},
        "right" : {"img_pyramid": "../images/right_pyramid",
                   "mask_pyramid": "../images/right_mask_pyramid"},
        "model": {"type": "pol", "degree": 2},
        "mesh": {"row":3, "col":3},
        "minimal_nb_pixels_per_mesh": 100,
        "output":
            "a_multiscale_output"
    },
    "pandora2d":
        ["resolution_1_config.json", "resolution_2_config.json", "resolution_3_config.json"] // pandora2d pipelines for each processed resolution

}

Warning

There must be as many masks as there are images in the image pyramid. The masks must be the same size as the images in the pyramids.

As with images, the masks paths specified in the pandora2d configurations are overwritten by the masks contained in the pyramids specified in the multiscale key.

Hint

It is possible to run a_multiscale_pipeline_with_masks.json located in the pandora2d github repository for the 1.1.2.dev0 version.

Outputs and results

Outputs structure

The configuration given in the example above will result in the following tree structure:

output
└── a_multiscale_output
    ├── iteration_1
        ├── disparity_map
            ├── row_map.tif
            ├── col_map.tif
            ├── correlation_score.tif
            ├── validity.tif
            ├── attributes.json
            ├── report
        └── config.json
    └── iteration_2
        ├── disparity_map
            ├── row_map.tif
            ├── col_map.tif
            ├── correlation_score.tif
            ├── validity.tif
            ├── attributes.json
            ├── report
        ├── config.json
        ├── init_col_grid.tif
        ├── init_row_grid.tif
    └── iteration_3
        ├── disparity_map
            ├── row_map.tif
            ├── col_map.tif
            ├── correlation_score.tif
            ├── validity.tif
            ├── attributes.json
            ├── report
        ├── config.json
        ├── init_col_grid.tif
        ├── init_row_grid.tif

Note

The init_col_grid.tif and init_row_grid.tif files correspond to the initial disparity grids estimated in the previous iteration and used in the pandora2d pipeline of the current iteration.

Furthermore, disparity_map repositories and config.json files are pandora2d outputs.

Note

A MESH_validity band is added to the validity.tif files at each iteration, except the last one for which no new model is being estimated.

Example of inputs and outputs

If we launch the a_multiscale_pipeline.json file we obtain the following results at the last iteration:

Example of results at each iteration

Below are diagrams representing the results at each iteration of the multiscale mode.

To do this, we used image pyramids created with maricopa images with subsampling factors of 1, 2, and 4. We use 3 meshes in a row and 3 meshes in columns and a degree 2 polynomial model (arbitrary model choice). The pandora2d configuration is the one in a_segment_mode_pipeline.json.

For the right pyramid, we shifted the original maricopa left image by:

• 0 pixel on the first third of the image in rows

• 2 pixels on the second third of the image in rows

• 5 pixels on the last third of the image in rows

There is no shift in columns.

Then we cropped the top of the image to avoid having no data.

We can see that our latest disparity maps (iteration 3) clearly show:

• A 0-pixel shift in rows in the first third of the image

• A 2-pixel shift in rows in the second third of the image

• A 5-pixel shift in rows in the third third of the image

• A 0-pixel shift in columns across the entire image