@@ -12,46 +12,31 @@ Many commercial RGB cameras use a technology called Color Filters Array (CFA). T
...
@@ -12,46 +12,31 @@ Many commercial RGB cameras use a technology called Color Filters Array (CFA). T
Because of the filters the pixels on the sensor will only acquire one color (being red, green or blue). This leads to a raw acquisition that is a gray-scale image. See example bellow:
Because of the filters the pixels on the sensor will only acquire one color (being red, green or blue). This leads to a raw acquisition that is a gray-scale image. Example:
The goal of this project is to perform *demosaicking*: recover all the missing colors for each pixel. This way we will recover the full RGB image.
The goal of this project is to perform *demosaicking*: recover all the missing colors for each pixel. This way we will recover the full RGB image.
We propose you to reconstruct 4 images (you can find them in the `images` folder of this project). These images are from the open dataset of the **National Gallery of Art**, USA, which can be found [here](https://github.com/NationalGalleryOfArt/opendata). To reconstruct these images we provide you the forward operator, modeling the effect of a CFA camera. This operation is described in the file `src/forward_operator.py`.
We propose you to reconstruct 4 images (you can find them in the `images` folder of this project). These images are from the open dataset of the **National Gallery of Art**, USA, which can be found [here](https://github.com/NationalGalleryOfArt/opendata). To reconstruct these images we provide you the forward operator, modeling the effect of a CFA camera (either Bayer of Quad Bayer pattern). This operation is described in the file `src/forward_operator.py`.
## 3. How to proceed?
## 3. How to proceed?
All of the images to reconstruct have the same size (1024x1024) and are RGB. The patches to use are all same sized and squared (32x32) and are RGB too. To achieve this project you have to:
All of the images to reconstruct have the same size (1024x1024) and are RGB. You have to use methods seen during this semester to recover a fully colored RGB image using **only** the raw acquisition (gray-scale image) and the forward operator.
-**a.** Extract all possible pieces (32x32 patches) from the image to reconstruct, let's call a piece from the image to reconstruct a query (*q in the figure below*).
-**b.** Find for each query the patch (*p in the figure below*) that best fit the query with the help of an image processing based measure **that you will build (feel free to search on the internet!)**. The patches from the database must be used only **one time**.
-**c.** Reconstruct the image with the found patches.
## 4. Some hints to start
## 4. Some hints to start
We want you to find a metric that allows to find the patch that best fit a given query. Remember how to compute the distance between two images (BE2, BE3).
We provide in `src/demo_reconstruct` a basic interpolation image which can help you to start. You can also find in the academic litterature techniques (interpolation, inverse problem, machine learning, etc) that solves demosaicking.
You can look further and find among these references some other metrics that you can implement to have better results:
- Histogram in image processing [Wikipedia](https://fr.wikipedia.org/wiki/Histogramme_(imagerie_num%C3%A9rique))
*The time of computations might be very long depending of your machine. To verify that your method works, try to work with smaller test images.*
- Mean Square Error [Wikipedia](https://en.wikipedia.org/wiki/Mean_squared_error)
*The time of computations might be very long depending of your machine. To verify that your method works, try to work with a few patches number (look at the source code and the provided documentation with the help() function).*
*Don't forget: images are RGB! The three channels are important!*
## 5. Project repository
## 5. Project repository
The project has the following organization:
The project has the following organization:
~~~text
~~~text
image_processing_project/
sicom_image_analysis_project/
├─.gitignore # Ignore this
├─.gitignore # Git's ignore file
├─images/ # All images you need to reconstruct
├─images/ # All images you need to reconstruct
│ ├─img_1.png
│ ├─img_1.png
│ ├─img_2.png
│ ├─img_2.png
...
@@ -60,24 +45,27 @@ image_processing_project/
...
@@ -60,24 +45,27 @@ image_processing_project/
├─main.ipynb # The notebook to run all of your computations
├─main.ipynb # The notebook to run all of your computations
├─output/ # The output folder where you can save your reconstruction (ignore .gitkeep)
├─output/ # The output folder where you can save your reconstruction (ignore .gitkeep)
│ └─.gitkeep
│ └─.gitkeep
├─patches.npy # All of the patches from ImageNet saved in one file
├─README.md # Readme, contains all information about the project
├─readme.md # ReadMe, contains all information about the project
├─readme_imgs/ # Images for the Readme, ignore it
├─readme_imgs/ # Images for the ReadMe, ignore it
│ ├─example.png
│ ├─img0.jpg
│ └─patterns.png
│ └─img1.png
├─requirements.txt # Requirement file for packages installation
├─requirements.txt # Requirement file for packages installation
└─src/ # All of the source files for the project, your source file must appear here!
└─src/ # All of the source files for the project, your source file must appear here!
├─demo_reconstructions.py
├─demo_reconstructions.py # Example of demosaicking | **Do not modify**
├─reconstruct.py
├─reconstruct.py # File containing the main reconstruction fonction
└─utils.py
├─checks.py # File containing some sanity checks | **Do not modify**
├─forward_model.py # File containing the CFA operator | **Do not modify**
└─utils.py # Some utilities | **Do not modify**
~~~
~~~
You are expected to write your own code in new files of `src`, and call it in the `src/reconstruct.py` file.
## 6. Instructions:
## 6. Instructions:
Each group will submit its work in Chamilo, inside the work **Image_processing_projects_2023**. It must contain:
Each student will submit its work in Chamilo, inside the work **Image_analysis_projects_2023**. It must contain:
- A report **as a pdf file** with the name **name1_name2_name3.pdf**.
- A report **as a pdf file** with the name **name.pdf**.
- Your code as a archive (all of the provided project folder named image_processing_folder **except the `patches.npy` file**).
- Your code as a archive with the name **name.zip**.
The code and the report must be **written in English**.
The code and the report must be **written in English**.
...
@@ -87,36 +75,16 @@ Your report **must be a pdf file** written in English. In this file you are aske
...
@@ -87,36 +75,16 @@ Your report **must be a pdf file** written in English. In this file you are aske
1. The problem statement, show us that you've understood the project.
1. The problem statement, show us that you've understood the project.
2. The solution that you've chosen, explaining the theory.
2. The solution that you've chosen, explaining the theory.
3. With tools that **you've built** show us that your solution is relevant and working (*hint: maybe try to reconstruct an image with a patch database built with the image itself...*).
3. With tools that **you've built** show us that your solution is relevant and working
4. Results, give us your results of the images `img_1`, `img_2`, `img_3`, `img_4`.
4. Results, give us your results of the images `img_1`, `img_2`, `img_3`, `img_4`.
5. Conclusion, take a step back about your results. What can be improved?
5. Conclusion, take a step back about your results. What can be improved?
### 6.2. The code:
### 6.2. The code:
- Your code must compile and be operational.
- Your code must be operational.
- In the notebook, feel free to add cells to show us how your code works and to show us the tools you've developped to evaluate your work.
- In the notebook, feel free to add cells to show us how your code works and to show us the tools you've developped to evaluate your work.
- Comment your code when needed.
- Comment your code when needed.
## 7. Upload the whole project on jupyter hub
## 7. Supervisors
Jupyter hub allows to upload only files and not a folder organization. You can upload the project in one step by uploading a **.zip version** of the project on jupyter hub and unzip it with the python code below (you can write this code in a temporary notebook in jupyter hub):
