Instructor: Pinar Duygulu
Office :  EA 433
e-mail : duygulu[at]cs.bilkent.edu.tr
Phone :  (312) 290 31 43
Office hours:  by appointment..
Course web page: http://www.cs.bilkent.edu.tr/~duygulu/Courses/CS554/Fall2006
Textbook: Computer Vision - A modern Aproach  by David A. Forsyth & Jean Ponce, Prentice Hall, Ed. 1, 2002
Other textbooks:
Computer Vision by Dana Ballard and Chris Brown (available online)
Digital Image Processing by Rafael Gonzalez and Richard Woods
Computer Vision by Linda Shapiro and George Stockman
Related Material: http://www.cs.bilkent.edu.tr/~duygulu/CVlinks.html
Also other complementary articles that will be made available
Previous course materials are avilable at
http://www.cs.bilkent.edu.tr/~duygulu/Courses/CS554/Spring2006
http://www.cs.bilkent.edu.tr/~duygulu/Courses/CS554/Fall2004
http://www.cs.bilkent.edu.tr/~duygulu/Courses/CS554/Spring2004
Time & Location: Wednesdays 9:40-11:30, Fridays 11:40-12:30, EA 502
Course Description:  Basic concepts in computational vision. Relation to human visual perception. The analysis and understanding of image and video data. Mathematical foundations, image formation and representation, segmentation, feature extraction, contour and region analysis, camera geometry and calibration, stereo, motion, 3-D reconstruction, object and scene recognition, object and people tracking, human activity recognition and inference.
Prerequisites:Knowledge of linear algebra and calculus, probability and statistics
Topics:
Introduction,  Color and Light, Linear Filters, Texture,  Edge detection,  Interest Points, Cameras, Multi-view Geometry, Stereopsis,  Motion,  Segmentation,  Object recognition,  Face recognition,  Image and Vieo Databases
Grading:
Reading Assignments 10%
Programming Assignments 30%
Paper Presentations 15%
Term Project 40%
Participation 5%
  

---

Announcements:

• Grades are  finalized
  

---

Lectures

  

Introduction September 13 (slides)	Topics What is computer vision? Why is it difficult? Which cues do humans use to perceive? Application areas Links Vision in the news - CNN article Optical Illusions The joy of visual perception
Basics September 15 (slides)	Topics Review of Linear Algebra,Geometrical Transformations, Introduction to Matlab,Handling Images in Matlab Readings: Matlab primer (.pdf) by Kermit Sigmon, University of Florida Matlab tutorial and codes by Stefan Roth, Brown University Basic math (.pdf),  figures (.pdf) by Martial Hebert, CMU Links The Mathworks' "Getting started with Matlab
Image Processing September 20,22 (slides)	Topics Image representation, Point processing, Blob processing, Applications of binary image analysis,Thresholding,Connected component analysis,Mathematical morphology,Region propoerties Readings: Chapter 3 from Shapiro&Stockman Chapter 3 from Gonzales & Wood Links Morphological operators applet,
Linear Filters September 27, 29 (slides1, slides2)	Topics Linear filters, convolution, smoothing, derivatives, Fourier transform, sampling and aliazing, gaussian pyramids Readings Chapter 7 from Forsyth&Ponce Notes on masks, by Shapiro & Stockman Computer vision for interactive computer graphics,W. T. Freeman, D. Anderson, P. Beardsley, C. Dodge, H. Kage, K. Kyuma, Y. Miyake, M. Roth, K. Tanaka, C. Weissman, W. Yerazunis,  in IEEE Computer Graphics and Applications, volume 18, number 3, May--June, pp. 42-53, 1998. Links Tutorial on Convolution, by David Young
Edge Detection October 4 (slides)	Topics Derivatives, Edge detection, Hough Transform  Readings Chapter 8 from Forsyth&Ponce A Computational Approach to Edge Detection, J. Canny, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol 8, No. 6, Nov 1986. Chapter 4 from Olivier Faugeras' book: Three-Dimensional Computer Vision, MIT Press, 1993 Links Tutorial on Hough Transform, by David Young Hough line detection applet, Hough circle detection applet
Texture October 4 (slides)	Topics Texture analysis and synthesis  Readings Chapter 9 from Forsyth&Ponce Pyramid based Texture Analysis/Synthesis, David Heeger and James Bergen, SIGGARPH 1995 A Computational Model of Texture Segmentation, J. Malik and P. Perona, Proc. Computer Vision and Pattern Recognition, 1989 Eraly Vision and Texture Perception, J.R. Bergen and E.H. Adelson, Nature, 1988 W.Y. Ma and B.S. Manjunath, Texture features and learning similarity, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, pp. 425-430, June, 1996 Texture Synthesis by Non-parametric Sampling, Alexei A. Efros and Thomas K. Leung, IEEE International Conference on Computer Vision (ICCV'99), Corfu, Greece, September 1999,  Image Quilting for Texture Synthesis and Transfer, Alexei A. Efros and William T. Freeman, Proceedings of SIGGRAPH '01, Los Angeles, California, August, 2001 Links Wavelet Tutorial by Robi Polikar
Interest Points October 6 (slides)	Topics Harris Detector, Local invariant points, SIFT descriptors Readings A combined corner and edge detector, Chris Harris and Mike Stephans, Proceedings of The Fourth Alvey Vision Conference, Manchester, pp 147-151. 1988 Local Greyvalue Invariants for Image Retrieval,  C. Schmid and R. Mohr. In Pattern Analysis and Machine Intelligence, 1997. Indexing based on scale invariant interest points. K.Mikolajczyk and C.Schmid. In International Conference on Computer Vision, 525-531, 2001 Distinctive Image Features from Scale-Invariant Keypoints, David Lowe, International Journal of Computer Vision, 2004. Links Tutorial on Recognition and Matching based on local invariant features, by Cordelia Schmid and David Lowe, at CVPR 2003
Radiometry and Color October 11 (slides1, slides2)	Topics Radiometry, measuring light, color perception, color spaces Readings: Chapter 4 and 6 from Forsyth&Ponce The foundations of color measurement and color perception by Brian A. Wandell
Student presentations October 13	by Muhammet Bastan, AnnoSearch: Image Auto-Annotation by Search, Xin-Jing Wang, Lei Zhang, Feng Jing, Wei-Ying Ma, CVPR2006 (slides) by Selen Pehlivan, Detecting Unusual Activity in Video, H. Zhong, J. Shi and M. Visontai, CVPR 2004.(slides)
Cameras and  Calibration October 18 (slides1, slides2)	Topics Perspective projection, pinhole camera model, lenses, camera geometry, camera calibration  Readings Chapters 1, 2 and 3 from Forsyth&Ponce An Introduction to Projective Geometry by Stan Birchfield A Flexible New Technique for Camera Calibration, Zhengyou Zhang,  Technical Report MSR-TR-98-71,  Microsoft Research, Redmond, WA 98052 Links Brunelleschi and the Origin of Linear Perspective Links related to Camera Calibration
Student presentations October 20	by Esat Belviranli, Object Level Grouping for Video Shots, J. Sivic, F. Schaffalitzky and A. Zisserman,IJCV Volume 67, Number 2 / April, 2006  (slides) by Hidayet Aksu, Object Detection by Contour Segment Newtworks, Vittorio Ferrari, Tinne Tuytelaars, Luc Van Gool, ECCV2006 (slides)
Multi view Geometry November 1 (slides)	Topics Epipolar geometry Readings Chapter 10 from Forsyth&Ponce Epipolar Geometry and the Fundamental Matrix,  in Multiple View Geometry in Computer Vision , Hartley & Zisserman, 2000 Links Tutorials on Multi-view Geometry by Zisserman, Hartley and Pollefeys
Student presentations November 3	by Mumin Cebe, Identifying Semantically Equivalent Object Fragments, Boris Epshtein Shimon Ullman,CVPR 2005 (slides) by Armagan Alper, Vehicle Segmentation and Tracking from a Low-Angle Off-Axis Camera , Neeraj K. Kanhere, Shrinivas J. Pundlik, Stanley T. Birchfield, CVPR June 2005 (slides) by Kardelen Hatun, Finding and Tracking People from the bottom up, Deva Ramanan and D. A. Forsyth, CVPR 2003 (slides)
Stereopsis November 8 (slides)	Topics Stereopsis, Matching, Reconstruction  Readings Chapter 11 from Forsyth&Ponce Links Middlebury Stereo Vision page - good source for stereo dataset Stereo Images with Ground Truth Disparity and Occlusion, from University of Bonn A Matlab code  by Carlo Tomasi to generate random dot stereograms and to find correspondences with the Marr-Poggio algorithm. The art of Stereo Photography How to freeview Stereo Images  Modeling and Rendering Architecture from Photographs, by Debevec, Taylor and Malik
Student presentations November 10	by Selcuk Sandikci, Multi-scale gesture recognition from time-varying contours,  Hong Li; Greenspan, M., ICCV 2005 (slides) by Melih Kandemir, Discovering objects and their location in images, Josef Sivic, Bryan C. Russel,2 Alexei A. Efros, Andrew Zisserman, William T. Freeman, ICCV 2005 (slides)
Motion November 15 (slides)	Topics Optical flow, structure from motion, Tracking Readings An iterative image registration technique with an application to stereo vision, Bruce Lucas and Takeo Kanade, Proceedings of the 7th International Joint Conference on Artificial Intelligence (IJCAI), 1981 Detection and Tracking of Point Features.Carlo Tomasi and Takeo Kanade.  Carnegie Mellon University Technical Report CMU-CS-91-132, April 1991.    Good Features to Track, Jianbo Shi and Carlo Tomasi, IEEE Conference on Computer Vision and Pattern Recognition, pages 593-600, 1994. Feature based methods for structure and motion estimation, Phil Torr and Andrew Zisserman, in Vision Algorithms: Theory and Practice, B. Triggs, A. Zisserman, R. Szeliski (Eds.), Springer (2000) Links KLT : Implementation of Kanade-Lucas-Tomasi Feature Tracker Motion Flow in Computer Vision, by Jana Kosecka and Shankar Sastry Lecture on motion, by Robyn Owens Layered Image Representation demo by J. Wang Barber pole illusion Image sequences: Flower garden(.mov) A review on background estimation tecniques
Student presentations November 17	by Akif Burak Tosun, Multiple Paths Extraction in Images Using a Constrained Expanded Trellis, Changming Sun and Ben Appleton, PAMI 2005 (slides) by Alptug Dilek, A Non-Local Algorithm for Image Denoising, A. Buades, B. Coll, and J.-M. Morel, CVPR 2005 (slides)
Mosaics Noveber 22 (slides)	Topics Homographies, Image Mosaics Readings R. Szeliski and H.-Y. Shum, Creating Full View Panoramic Image Mosaics and Environment Maps, Proc. ACM SIGGRAPH, 1997, longer version: Panoramic Image Mosaics, Technical report, MSR-TR-97-23, 1997 M. Brown and D. G. Lowe. Recognising Panoramas. In Proceedings of the 9th International Conference on Computer Vision (ICCV2003), pages 1218-1225, Nice, France, October 2003. Planar Scenes and Homography lecture notes by Serge Belongie Links Autostitch: Automatic Image Stiching
Student presentations November 24	by Erdem Sahin, Adaptive View-Based Appearance Models, Louis-Philippe Morency Ali Rahimi Trevor Darrell, CVPR 2003 (slides) by Alexander Suhre, Automatic face recognition for film character retrieval in feature-length films, O. Arandjelovic, A. Zisserman, CVPR 2005 (slides)
Segmentation November 29 (slides)	Topics Segmentation, Grouping, Fitting Readings Normalized cuts and image segmentation, Jianbo Shi and Jitendra Malik, IEEE Transactions on Pattern Analysis and Machine Intelligence(PAMI), 2000 a shorter version  published in IEEE Conf. Computer Vision and Pattern Recognition(CVPR), June 1997, Puerto Rico  Laws of Organization in Perceptual Forms, Max Wertheimer, first published as Untersuchungen zur Lehre von der Gestalt II, in Psycologische Forschung, 4, 301-350. Translation published in Ellis, W. (1938). A source book of Gestalt psychology (pp. 71-88). London: Routledge & Kegan Paul. Learning to Detect Natural Image Boundaries Using Local Brightness, Color, and Texture Cues, D. Martin, C. Fowlkes, and J. Malik.   to appear in PAMI  Links Computational Models of Perceptual Organization, talk slides by Stella Yu Laws of Organization in Perceptual Forms, Max Wertheimer The Berkeley Segmentation Dataset and Benchmark Matlab Code for k-means and Expectation Maximization by Carlo Tomasi A tutorial on Clustering Algorithms with a K-means clustering applet
Student presentations December 1	by Orkun Tuncel, Fast Multiple Object Tracking via a Hierarchical Particle Filter, Changjiang Yang, Ramani Duraiswami and Larry Davis, ICCV 2005 (slides) by Mehmet Kok, Detecting Irregularities in Images and in Video", Oren Boiman, Michal Irani - 10th ICCV, 2005 (slides)
Recognition December 6 (slides)	Topics Model based and template matching based methods for recognition Readings , Object Detection using statistics of parts, Henry Schneiderman and Takeo Kanade, International Journal of Computer Vision Robust Real-time Object Detection, Paul Viola, Micheal Jones, International Journal on Computer Vision, 2001 Recognition by linear combinations of models, S. Ullman and R. Basri, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.13, No.10, October 1991 Shape Matching and Object Recognition Using Shape Contexts , Serge Belongie, Jitendra Malik and Jan Puzicha, IEEE Transactions on Pattern Analysis and Machine Intelligence(PAMI), 24(4):509-522, April 2002. Object Class Recognition by Unsupervised Scale-Invariant Learning, Rob Fergus, Pietro Perona, and Andrew Zisserman, Conference on Computer Vision and Pattern Recognition, (2003). Face recognition using eigenfaces, M. Turk and A. Pentland, Proc. IEEE Conference on Computer Vision and Pattern Recognition, Maui, Hawaii, 1991 Face Recognition Across Pose and Illumination R. Gross, S. Baker, I. Matthews, and T. Kanade, Handbook of Face Recognition, Stan Z. Li and Anil K. Jain, ed., Springer-Verlag, 2005 Links International Object Recognition Workshop, Sicily, 2004 The PASCAL Visual Object Classes Challenge
Student presentations December 8	by Tuncer Erdogan, Building models of animals from video, Ramanan D, Forsyth D.A., Barnard K., PAMI 2006 (slides) by Zeynep Yucel, What went where [motion segmentation], Wills, J.; Agarwal, S.; Belongie, S., CVPR 2003 (slides)
Image and Video Databases December 13 (slides)	Topics Retrieval, browsing and other novel applications on large datasests Readings The Earth Mover's Distanceas a Metric for Image Retrieval, Y. Rubner, C. Tomasi, L.J. Guibas, Technical Report STAN-CS-TN-98-86, Computer Science Department, Stanford University, September 1998 Faces and Names in the News Tamara Miller, Alexander C. Berg, Jaety Edwards, Michael Maire, Ryan White, Yee Whye Teh, Eric Learned-Miller, David A. Forsyth, CVPR 2004 Links TRECVID - TREC Video Retrieval Evaluation Blobworld - Image retrieval using regions
Student presentations December 15	by Erkan Okuyan, Edge detection to guide range image segmentation by clusteringtechniques, Bellon, O.R.P.   Direne, A.I.   Silva, L. m ICIP1999 (slides) by Eyuphan Bulut, Geometric Context from a Single Image, D. Hoiem, A.A. Efros, and M. Hebert,ICCV 2005 (slides)
Student presentations December 20	by Mehmet Cihan Ozturk, Video Google: A Text Retrieval Approach to Object Matching in Videos", by Josef Sivic and Andrew Zisserman, ICCV2003 (slides) by Mustafa Kalkan, Improving Web-based Image Search via Content Based Clustering, N.Ben-Haim, B. Babenko, S. Belongie, CVPR 2006 (slides) by Ismet Zeki Yalniz, Recognizing Action at a Distance, A.A. Efros, A.C. Berg, J. Malik, ICCV 2003 (slides)
Student presentations December 22	by Hakan Tuna, "Real-Time Non-Rigid Surface Detection.",J. Pilet, V. Lepetit, and P. Fua (slides) by Hakan Uraz, "Detecting Pedestrians using Patterns of Motion and Appearance", Paul Viola, Michael J. Jones, and Daniel Snow, ICCV 2003, by Hakan Uraz (slides)

---

Assignments:

• Reading Assignment #1:
• "Blobworld: Image Segmentation Using ExpectationMaximization and Its Application to Image Querying", C. Carson, S. Belongie, H. Greenspan and J. Malik,  IEEE Trans. PAMI, pp. 1026-1038, 2002
  
• Due: September 20
• Reading Assignment #2
• Distinctive Image Features from Scale-Invariant Keypoints, David Lowe, International Journal of Computer Vision, 2004.
• Due: October 6
• Reading Assigment  #3
• Face recognition using eigenfaces, M. Turk and A. Pentland, Proc. IEEE Conference on Computer Vision and Pattern Recognition, Maui, Hawaii, 1991
• Due: December 6
  
• Programming Assignment #1
• Due:October 20
  
• Texture Analysis
• Design spot and bar filters in different orientations
  
• Build a Gaussian Pyramid of four scales
• Apply your filters on different scales, obtain responses to your filters
• Use mean and variance to compute statistics of filter responses
• Compute the similarity of two textures based on the Euclidean distance of the statistics
• Download Brodatz Texture Album
• For each of the images in the album, divide the image into 4x4 grids to obtain 16 patches
  
• Use one of the patches as the query and rank the other patches (of all the images) according to the similarity criteria defined above
• Report the results
• Your report should include
• The filters that you designed: how you designed them, the parameters that you used, how they look like, their response to some images
  
• Take 10 images from the database, and one patch from each. Show the top 16 images with the highes rank
• Evaluate the performance of your system
• For all the images
• choose one patch, and rank the most similar ones. Look at the first 16, and count how many of the other patches correspond to the same texture
• report the result
• Report the average performnace.
• Discuss the results
• Compare your results with one of the studies on the same database
  
• give reference, the method of that study and the results of that study
  
• Submit your report together with a tar ball of your codes (do not include the images) through
  
• http://pclinuxserver.cs.bilkent.edu.tr/cgi-bin/submit/submit.cgi/duygulu
  
• Reference: Preattentive texture discrimination with early vision mechanisms, Jitendra Malik and Pietro Perona, Journal of Optical Society of America A, 7(5), May 1990, 923-932

• Programming Assignment #2
• Due: December 6
  
• Action Recognition
• You are required to recognize actions performed by different people on static backgrounds
• You will use the dataset provided in http://www.wisdom.weizmann.ac.il/~vision/SpaceTimeActions.html#Database
• The dataset is in avi format. In order to obtain the image sequence you may use the following
  
• http://www.virtualdub.org/
• First, find the silhouettes by a background subtraction method
• Modeling the background as Gaussian distribution
• Subtract each frame from the background model
  
• Since background subtraction may result in many noise pixels apply morphological operators to remove the noise (Optional: If you still have noise pixels try to apply hystheresis thresholding)
• Then obtain the bounding boxes for the silhouette images by finding the extremum points
  
• If you cannot obtain good silhouette images, for the following you may continue with the masks provided in the website (If that is the case, clearly state it)
• Divide the bounding boxes into NxM grids
• Represent each grid with the following feature: (Number of black pixels in the grid) / (Total number of pixels)
• Create a training set consisting of two examples from each action
• Put the remaining examples into the test data
• Then, for each test example find the matching action from the training set
  
• Compare the test example with each of the training examples using a voting scheme 
  
• For comparison all frames of the test example should be compared with all the frames of the all training example
• Take a frame and find the matching frame among the training examples as the one with the minimum distance
  
• The action in the training set with the maximum number of matching frames will take the highest vote
  
• Finally report the results.
• Your report will include:
• Explanation of the problem and your strategy to solve the prolem
• Explanation of background subtraction with listing of related parameters (mu and sigma of  the background models), k in thresholding
• Explanation of morphological operators you applied, with description of the structuring element
  
• Example image sequences and results of background subtraction for those images (For each action give at least one example) (before and after application of morphological operators)
  
• For each action the performence score (true matches/total number of examples for that action)
  
• A confusion table showing how the actions are mixed with each other
• Discussion of the results
• Bonus:
  
• The strategy explained above is very simple, and does not consider the temporal information.
  
• After representing the individual frames as grids, if you can find a better representation to consider the temporal information you will get a 20% bonus (You can implement a related paper, if that is teh case do not forget to give a reference)
• 
  
• Programming Assignment #3
• Due: december 22
• Creating Panoramas
  
• Create panaromas similar to the examples shown in the class. The method will be similar to Autostich but will be simpler.
• In particular, implement the following:
• Take an image pair
• Use Lowe's SIFT operator to extract and describe the interest points from the images
• Apply RANSAC to find homogrphy (randomly choose 4 points, find the homography matrix. apply H to all points, find the errors, put a threshold to count inliers and outliers, choose the best points which give the maximum number of inliers)
• Warp one of the images onto the other image (create a new black image, plot one of the images, for the points on the other image compute the transform according to the homography matrix, if there are gaps fill them by looking at the neighbor points, if there are some overlaps, either choose randomly or take the average)
• For each image pair create the following images showing that your algorithm is working for those steps:
• Show the interest points on the images
• Show the matches according to Lowe's operator
• Show the points producing the best homography and show the homography matrix
• Show the matches according to the homography
• Show the inliers and the outliers of the best match after finding the homography
• Warp the images so that the images will be shown on top of each other
• Write a report to explain the method that you use and present some example results (as stated above) to show that your algorithm is working
• Reference paper:  M. Brown and D. G. Lowe. Recognising Panoramas. In Proceedings of the 9th International Conference on Computer Vision (ICCV2003), pages 1218-1225, Nice, France, October 2003.
• Image set:
  Use the images of Autostich. You can obtain the whole set of images from
  http://www.cs.ubc.ca/~mbrown/panorama/testimages.zip
  You do not need to find he matches automatically. Choose the matching image pairs by hand.
• Also use the images in http://campar.in.tum.de/view/Chair/TeachingWs05ComputerVision
  

---

Policies

Important notes about evaluation:

    Assignments:
        There will be three reading assignments and three programming assignments      
        Late homeworks are not accepted     
        All programming assignments are due midnight and will be sent by e-mail
            In your e-mail use the following format in the title
            CS554 - Programming assignment #
            Your programming assignmenments should be sent as a tar ball in the following format
                <name_surname_PA_#>.tar
        All reading assignments are due before the lecture hours and will be given to the instructor as printed out
            Reading assignments will be summary of the given paper. It should be about one page,
            and you should explain the main contributions and the important points of the proposed
            methods in your own words. Do not include any figures or formulas. Do not write the values
            of parameters or thresholds unless they are very important. Assume that you are submitting
            one or two pages summary of your paper to a conference.

    Projects 
    The project may be
        An original implementation of a new or published idea,
        A detailed emprical evaluation and comparison of the existing implementations of two or more methods,     
    You will work in groups of two or three
    You are required to write a proposal, progress report, final report, and do a demonstration and a presentation 
        Project proposal will be a short description of the problem you would like to tackle, objective of the study,
         proposed algorithms, hardware/software tools and data that you plan to utilize, and evaluation strategies
        that you plan to use. Also provide a short list of related references.
        Progress report will describe your progress in the project and your plans for the rest of the semester
        Final report will be a well-written report which provides proper motivation for the task, proper citation
        and discussion of related literature, proper explanation of the details of the approach and implementation
        strategies, proper performance evaluation, and detailed discussion of the results. You should highlight
        your contributions and conclusions.
            Final report guidelines:
            Follow IEEE two-column format as shown in the example and the format definition table and glossary.
            The page limit is 6 pages.
            The report should not have any page numbers, headers or footers.
            You can use IEEE's LaTeX template or Word template. (LaTeX users: Be sure to use the template's conference mode.)
            PDF submission is recommended.
        Presentation will be in the form of poster session and each team will show their contributions on a poster
        which will fit to a board of approximately 1m X 1m
       
    Presentations:
    Your presentations will be evaluated according to the following criteria. Please, consider them in preparing your presentations:
        Understanding of the topic - how confident are you with the paper that you present
        Review of the related work - not just mentioning but by reading some of them to understand and relate to your paper
        Giving an overview of the paper -  the main contributions of the paper, and an overview of the approach
        Explaining the details - understanding and explaining the formulas and methods given in the paper
        Presentation - in general how well you are prepared to give the talk
        Use of visual material when available