CS5150 Software Engineering Milestone 3 Report Make3D Project November 5th, 2010 Diego Accame Karen Ho Thomas Petracca

a Kohei Shinkawa Supasorn Suwajanakorn Goran Svorcan Glenn Ulansey I. General Summary The purpose of this project is to develop code that can successfully convert 2D images into 3D models. The project uses Professor Ashutosh Saxenas Make3D algorithm, currently written in MATLAB, and it is our intent to convert the code to C++ making use of several open-source libraries, including the OpenCV library. The final product will also include a command line interface which will allow for easy use by non-technical users. II. Functional Goals and Requirements Our requirements since the previous report have changed as a result of a better understanding of the challenges in converting the MATLAB code as well as a better understanding of our own skills in MATLAB and C++. We originally defined success as consisting of three parts. First, the complete implementation and conversion of the MATLAB Make3D code into C++ (including testing to ensure that the output from both was identical). Second, the modularization of the code so that individual functions could be executed and tested independently. And finally the implementation of a user interface for creating 3D images. While we still have these three requirements as our goals, their contents have changed since our last report to the following: Code Conversion: We are still aiming to convert as much of the MATLAB code into C++ as possible, however it is our top priority to convert at least the first third of the code (which does most of the heavy number crunching), and ensure that it is properly tested (meaning its results match up with the MATLAB code) and is not too computationally intensive. We reached this decision after discussing our progress with our client in order to determine a more realistic goal. Modularization: We have already broken up the Make3D code into chunks based on the MATLAB functions involved and have been testing each function separately against the MATLAB code as we implement them. This not only allows us to minimize the potential for errors but also permits a user to run just a subset of the full functionality. This is useful if the user wishes to check the results using different input parameters. The main change to this requirement has been an organizational one. During a meeting with our client we decide to restructure our code with a more data focused model in mind. This would mean grouping functions that work on similar sets of data together in one class along with that data allowing for easier access to it and removing the risk of creating multiple and differing copies of the data.

UI: While a sophisticated UI has never been our goal, we will create a basic command-line application, which will allow the user to provide an image as input and returning the generated data. III. Documentation Documentation continues to be a major non-functional requirement of our project, and one we work on concurrently with our code development. The purpose of this documentation is to improve code consistency and organization, and to help future developers understand what weve written. We are still keeping with our Doxygen format for function declarations and descriptions, which provides us with a simple and clean view of each function in various formats. We also include further documentation in our function definitions which consists of technical descriptions of variables and sections of our code, however these comments do not include many descriptions of what the code is actually doing. This is largely because we dont fully understand the different aspects of the Make3D algorithm as learning it is not part of the requirements and would take too much time. However, after meeting with our client we decided that some brief descriptions of functionality would be useful and thus during our code review sessions with our client we will also work on adding these types of comments into the code. IV. New data oriented style We restructured our code per our clients request, to focus on the data instead of the functions that work upon them. Originally we had an overarching namespace, Make3D, that contained functions corresponding to each of the MATLAB files we were converting. Outputs from certain functions would then act as the inputs to others. We have now chosen to split the functions into classes by the data they work on. Each of these classes has a number of member variables that represent the key data structures for that class. This way, the data structures are easily accessible if someone wants to get specific input or output. Currently, this restructuring is still taking place and will be finished in the next few days. V. Previous problems In our previous presentation, we discussed a number of issues that had come-up, and our proposed solutions. The following is a summary of these issues: Lack of familiarity with the code: The evaluation we performed in the feasibility study was not accurate due to a gross underestimated of the amount of work and time we would need to devote to this project. We found out that MATLAB code is far denser and complex than we initially thought and that one line of MATLAB code translates roughly into ten lines of C++. Different skill sets: Different members of the team have different amounts of experience with MATLAB and C++. While some members may understand MATLAB code easily, they may not be as fluent in C++ and vice versa. This lead to team members taking longer than expected in different parts due to inexperience. Team approach: We split up the parts we wanted to accomplish the first few weeks and then created groups consisting of 1-2 team members to complete each of the parts. These groups would then divide the work between its members and then each member would code separately

and join their parts at the end. However, because of this separation, there was a significant delay in response to requests for help. This resulted in a considerable amount of wasted time. Poor scheduling: Our initial schedule we created was quite vague, and only the few main sections of the project were mentioned. The proposed schedule was too optimistic and most parts were not completed on time. This lead to people who had dependencies on some other piece of code having to wait for a long time until the other groups were done. As a result, we actually achieved far less than we intended for Milestone 2. Ineffective use of resources: The code produced by Professor Saxena was written in MATLAB with the Linux operating system in mind, and is thus more complicated to run in other operating systems. Most of our team members have Windows or Macintosh computers and not everyone has MATLAB installed. This made it extremely difficult for the team to develop new code and compare it to the existing code. Unorganized architecture: Because everyone was coding separately, each was producing code with vastly different styles. Some commonly used functions were also often written more than once, both within and between groups. This lead to very inconsistent code and not much thought was actually put into how to join the different parts together later on. The lack of any kind of architecture also made it hard to share commonly used functions as one person would have to search through everyone elses entire code in order to find what they needed. Lack of communication with the client: The client felt that there was very little communication between him and the group. He wanted more interaction between the team members and him. VI. Solutions After considering these problems, we came up with the following solutions that have been extremely effective: Reorganizing the architecture: We have completely reorganized the way our project is structured to make it easier to code and reuse common functions. Every MATLAB file (*.m) will have its own corresponding source file in C++. Furthermore, all of these will be within the same namespace. Two additional files were created in order to help with this organization. The first one is Make3DFunctions.h, which is the header file for all function declarations. This way we can know how to call and what to expect from each others functions without having to wait until their code is finished. The other file is Make3DUtil.h/Make3DUtil.cpp, which contains the declarations and the actual implementations of all helper functions. Having the files organized in this way makes it much easier to share code and eliminate the problem where a function is written more than once. Furthermore, we are documenting what functions and files are currently being worked on wiki page on our SourceForge account. We also decided on a set of coding conventions so that our code will be consistent. Peer programming: Because we were finding many problems with working separately, we have adopted the new method: peer programming. Each file/function is being worked on by groups of 2, where both team members will sit together and code on one computer. While it may seem less efficient, the quality of code produced when working together is far greater than that achieved by working alone. With two people coding on the same computer it is easy to spot the other persons mistakes and this allows for cleaner code and significantly reduces the time spent

debugging. Furthermore, because several people have different skill sets, working together makes people complement each other and therefore they can apply a broader set of knowledge to tackle the same problem. By working together we have greatly increased our productivity. Updated feasible schedule: After having a rough idea of the complexity of the code and the structure of the whole program, we were able to produce a new feasible schedule. The schedule is now split by individual files and around two members work on a given file at the same time. Visibility: We have been trying to communicate more often with our client. We had a meeting with him to talk about the organization of our project and every group will individually meet with the client on the following weeks to go over the code and make sure it is up to the standards expected by him. VII. Obstacles and Difficulties One of the major problems we ran into initially had to do with using MEX (a way to call C++ functions from MATLAB) to test our generated C++ code. MEX originally seemed to be exactly what we wanted as we could take data generated by MATLAB at certain points in the code and pass it into our C++ code, thus skipping the part of the MATLAB code that we replaced, and then seeing if the results matched. However, we quickly found that this approach would not be so simple. Most of us have had trouble getting MEX to work even at a base level, and have ended up spending more time on it rather than working on our code. That is why we decided to not make MEX testing an explicit requirement as it did not make testing any easier or efficient than simply printing out the results from MATLAB and our C++ code and writing a script to compare them. Scheduling has also been a source of trouble. While we were able to create a more reasonable and revised schedule we've found that some file conversions are falling behind. We still see our goals as achievable, but due to our varying skill levels and understanding of the actual Make3D algorithm, delays have occurred. VIII. Current State and Progress Using the solutions mentioned in the previous sections has greatly helped our understanding of MATLAB and OpenCV functionalities. As a result, our production rate has increased and we have converted substantially more MATLAB functions into OpenCV than we did in the previous phase. In the past two weeks, we have fully converted 20 additional helper-functions into C++ and polished the existing ones. We also installed two free libraries to save time writing our own version of code. Overall we were able to fully convert 8 out of 18 files (for a total of 10 fully converted files) for this milestone as opposed to fully converting only 2 for our previous milestone. We have made both internal and external reconfigurations to improve our programming environment and successfully increased our production rate. There are a few files we could not manage to convert to meet with our new schedule plan. Yet, as some of our members have already completed their tasks, it is possible to redistribute the tasks and catch up on the goal we set for our new requirement set. IX. Results We have successfully converted a large part of our files and have tested them thoroughly. Our functions

tend to be faster than their equivalent MATLAB functions, which can be easily seen in Fig 1. Most of our functions give exactly the same results as their equivalent in MATLAB. For those that dont give identical results, the results are very similar (they differ in small range which is acceptable from our clients view). For example, if we use Fig 2 as the input for the generateSuperPixel function and its equivalent in MATLAB, we get the pictures in Fig 3. We can see that they are very similar (the colors used to paint every superpixel are generated randomly and therefore one should compare the boundaries of every segment instead of looking at the colors) and if we inject this result into MATLAB, the final 3D model created is almost identical to the one created by the original algorithm. MATLAB File / Source File MATLAB time (s) 36.4 75.3 18.6 15.3 9.2 5.5 1.9 C++ time (s) 7.9 36.6 13.1 15.2 13.4 0.5 0.1

f_sup_old.m / calcSuperPixelFeature.cpp GenerateRay.m / generateRay.cpp gen_Sup_efficient.m / generateSuperPixel.cpp (Image size ~ 2000x2000) gen_Sup_efficient.m / generateSuperPixel.cpp (Image size ~ 1000x1000) gen_Sup_efficient.m / generateSuperPixel.cpp (Image size ~ 600x600) calculateFilterBanks_old.m / FilterBank.cpp gen_predicted_GS_efficient.m+genFeaVectorNew.m / GeneratePredictedGS.cpp + FeatureVector.cpp Fig 1: Time benchmarks table

Fig 2: Sample input image

Fig 3: Output of generateSuperPixels in C++ (left) and MATLAB (right)

X. Updated Schedule and Plan Since the last milestone, we have reworked with our programming style to be more peer programming oriented to utilize every team members ability and to cover the different levels of experience with the different technologies. Our progress speed is more promising than before to complete our new requirement on time. We have only one final milestone left which is December 1st (Milestone 4) with the complete coding, testing, and documentation of components 1-3 Below is our updated schedule until the end of project. Note that all activities except those noted are MATLAB functions or files converted or to be converted.

ID 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

Task Name

Oct 17, '10 Oct 24, '10 Oct 31, '10 Nov 7, '10 Nov 14, '10 Nov 21, T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M Glenn SetupDefault_New.m Glenn,Karen gen_Sup_efficient.m Glenn segmentImgOpt Glenn,Karen premergAllsuperpixel_efficient.m Diego,Gle GenTextureFeature_InnerMulSup.m Supasorn calculateFilterBanks_old.m Glenn,Karen gen_TextSup_efficient_GlobalH2.m Goran[20%],Tom AbsFeatureGenMex_MemoryEfficient.m Goran,Tom makeSubSupMaskNew2.m Goran,Tom analysesupinpatch.m Glenn,Karen SparseAverageSample2DOptimized.cpp Supasorn OldBatchVersion/makeRelativeFeatureVector Diego GenerateRay.m Kohei f_sup_old.m Supasorn gen_predicted_GS_efficient.m Supasorn genFeaVectorNew.m Diego,Kohei,Supasorn CleanedSupNew.m Goran,Tom gen_predicted.m Diego,Glenn,Goran,Karen,Kohei,Supasorn,Tom New Function definitions and List of

23

Matlab Functions Milestone 2 Presentation/Report Create/restructure file structure Testing, code review/cleanup, documentation, buffer room, thanksgiving break Re-structure code -> "Data-Oriented"
Task Split

Diego,Kohei,Goran Kohei

Supasorn,Kohei,Glenn,Kare External Milestone Inactive Task Inactive Milestone Inactive Summary Manual Task Duration-only Page 1 Manual Summary Rollup Manual Summary Start-only Finish-only Deadline Progress

Project: Make3d Schedule Date: Wed 11/3/10

Milestone Summary Project Summary External Tasks