Homework I
Edge adaptation and data aggregation
Many computational systems provide a logging service so system administrators can keep tabs on a system’s activity as well as diagnose problems based on knowledge gleaned from the logs. Continuous and automated processing log entries can speed up the identification of problems as well as the process of implementing corrective measures. Web servers are examples of such systems. A web server log contains information regarding HTTP-based accesses to web pages hosted by the server, including the Internet Protocol (IP) address of the host accessing the page, the time of a particular access, the number of bytes sent by the server, and the specific page addresses that have been served. For real examples of web server logs, consult the Internet Traffic Archive.
- Write an application that processes the logs produced by a web server. For each unique IP, compute the total number of bytes served to that IP address.
- Extend your application such that the log entries enter into the system as a stream, rather than being read from a file as fast as possible. You can replay the log entries based on their timestamps for this purpose.
- Every time a log entry regarding an IP address is received, update the total number of bytes served to that IP address and output the new value. You should discard IP addresses for which no activity was detected during the last hour.
- Further extend your application with an additional stream whose individual tuples contain queries. Each query tuple contains a k value. Every time a query is received, the top k IP addresses with the highest number of bytes served should be output into a result stream.
Use either Storm or Streams.
Deliverables
- Please put the code for each part into a separate project (or directory).
- Tar/gz or zip the contents of your homework in to a single file.
- Email me (bgedik@cs.bilkent.edu.tr) the homework. Do not include binaries or jar files. If you have data files, please just put a small sample of it.
- Please include a short write-up explaining your implementation (1-2 pages).