Project description

The course project will involve the design and implementation of an automated mix-n-match, part assembly based 3D modeling method plus a learned plausibility scoring scheme to evaluate the quality of the generated 3D models. See an example of a mix-n-match part assembly below:

• Training set: a set of segmented and labeled 3D chair models. The training set will be employed to train your plausibility scoring function. The models will be selected as a subset of ShapeNet and PartNet.

• Test input: Your mix-n-match assembly method will take as input a small set of pre-segmented and labeled 3D models (e.g., 3, 5, 10), which come from the training set. Early in the project, we will provide you with sample test input sets for you to experiment with. However, your method will be ultimately evaluated on new test inputs which will only be revealed to you in a short time span right before the project defence date (Monday, December 13). During that short time span, every project team must run their method on the new test inputs and submit the outputs to your TA Qimin. More details about this test mechanism will be provided later.

• Output: Given a test input set, your modelling method will return an ordered list of new 3D chair models, formed by taking parts from the input models and re-composing them. The order will be based on how plausible the models are as chairs based on the learned scoring function.

• Evaluation criteria: The composed 3D models will be judged foremost by their plausibility. That is, the new composition should be as "chair-like" as possible. Note that being plausible also implies that the parts should be well connected, like real chairs, but only in appearance, not necessarily physically in terms of the model being a single piece and watertight. Clearly, the ultimate test of plausibility for models generated by your method will not be based on your own scoring function; it will be based on an independent scoring function and/or human judgement, via a user study.

  In addition, the new model should also ideally be novel and "interesting": the more dissimilar it is from the models in the input test set, while still being plausible, the better. Models generated should be a true composition, from different shapes in the input set, not a replica of one of the whole models in the input set. The test of novelty will mainly be based on human judgement.

1. A Parser which takes as input a set of pre-segmented and labeled 3D models and converts them into an internal representation for further processing, i.e., mix-n-match modeling.

2. A Mixer which produces one or more new 3D models by recomposing parts, possibly geometrically transformed or deformed parts, from models in the input test set. This component has the most room for cleverness. It involves finding the most appropriate parts and sets of parts to be re-composed, as well as applying proper geometric transformations, deformations, and part connection mechanisms to ensure that the mixed models possess sufficient degrees of plausibility. A random mix-n-match without any part transformation will be a base line but will not receive a high grade. Surprising new chairs are more likely to emerge if more drastic part transformations are possible. Your design and implementation will be judged by its effectiveness and thoughtfulness.

3. A Scorer which takes a new 3D chair model and returns a plausibility score to judge its likeliness of being a chair. In this project, you are asked to implement a classifier, which is trained on data derived from a set of 3D chair models (the training set). Specifically, the classifier should be probabilistic instead of merely returning a binary label, i.e., yes/no with respect to being a chair or not. Given a new 3D model, the probabilistic classifier returns a numerical value, the plausibility score, which represents the likelihood that the model belongs to the chair category.

   There are a lot of online resources covering probabilistic classifiers, e.g., Wiki, this course notes from Prof. Mark Schmidt at UBC, and this online tutorial. You are free to learn and implement any classifier for the task at hand. Available tools and implementations can be utilized, but please make sure that you provide clear citation and explain what your group did to adapt the tool/implementation to the plausibility scoring.

   If you want a direct example of learning plausibility scores for 3D shapes to start with, please read Section 4.1 of the SIGGRAPH 2017 paper by a GrUVi Almumni, Chenyang Zhu. You can work from the provided plausibility scorer (available at this google drive link (1.2 Gb)) and improve upon it. The provided baseline scorer was inspired by the one described in Chenyang's paper, but it uses LeNet's architecture for each individual classifier.

   There are plenty of online tutorials for image classifiers using Convolutional Neural Networks, like this one. Further, there are opportunities to improve both the classifier architecture and to improve the dataset itself (read the augmentation session on the tutorial). Still, you should note that the provided scorer is a baseline and not the most advanced probabilistic classifier given today's standard in deep learning.

Go back to top

Evaluation:

The grade (35% of course grade) of your project will be based on three components:

• Quality of final project presentation and demo (10%) to be given on the project defense data (TBA).

• Quality of the final modeling results (10%), which is at the discretion of the TA and instructor. High-quality chairs generated ideally should be plausible, novel, and interesting/surprising. It must be a mix-n-match from parts in the input models; it cannot be an exact replica of an input model.

• An evaluation of the effectiveness, efficiency, and thoughtfulness of the mix-n-match scheme and the plausibility scorer designed and implemented (15%).

Go back to top