=========================================
==== Phishload Phishing Test-Database ===
=========================================

Creator: Max Maurer - University of Munich
Website: http://www.medien.ifi.lmu.de/team/max.maurer
Date: 2012-03-22
Number of URL in the DB: 11215
Number of non-phishing URLs: 1185
Number of assigned phishing pages: 3718 


=== WHAT IS THIS? ===
The phishload test database is a set of visited phishing websites and original websites that have been visited and scanned for testing purposes. Over a period of several weeks current phising links from Phishtank (http://www.phishtank.com/) have been collected and then been visited by a controlled instance of Firefox browsers.

Besides this the 1000 most popular websites derived from the ALEXA ranking (http://www.alexa.com/) have also been scanned as a baseline for non-fraudlent websites. In case a original website for one phishing website did not exist in that list it was manually added afterwards. So for each phishing website the original website is present in the set of non-fraudluent websites with more non-fraudulent websites that do not have any phishes. 

For each page three screenshots were taken. A screenshot of the whole browser instance as visible onscreen. A screenshot of the whole rendered website with all contents even if they were below the currently visible window portaion. And finally a screenshot cropped to the currently visible window region. We tried to maintain the inner width and inner height of the browser window to make the cropped screenshots easily comparable.

Besides the screenshots other parameters like the HTML-Content of the website and the status code was saved to be able to process it afterwards.

Each downloaded phishing website was manually assigned to a parent website in case one existed. If the website was not anymore a phishing website it was sorted to different categories.

=== WHAT IS THIS SET FOR? ===
This set can be used as a testing set for ANTI-Phishing software that tries to detect phishing websites either bei HTML-Content or by other means. With the assignment to their original pages it is possible to measure true an false positives for the whole set.

=== WHAT DO THE DB-FIELDS MEAN? ===
The DB consists of three different tables: websites, states, screenshots.

A deteailed exaplanation on the tables is found at the end of this document.

=== WHAT ELSE DO I HAVE TO KNOW? ===
The number of original websites is not as high as it perferrably should be. The phishing websites have only been assigned to one website of a brand that is somehow related to that phishing website. It is still possible that another part of the assigned website would be more similar to the phishing website as it is the case now. 

The rendering was done using a german ip-adress. Thats why pages that show commercials or similar things may show german content. Where possible we tried to use urls matching the given language for the phishing website.

=== DISCLAIMER ===
Use the set as you like but please give me credit where possible and tell me what you used it for. Besides there is no guarantee that every website has been correctly assigned and that every assignment makes perfect sense. When assigning such a large number of websites manually some errors will always occur. Please be aware that some of the captured websites may contain offensive or explicit content.

=== TABLE: States ===
Each phishig website has been assigned a state when manually verifying it. This state is used to categorized websites that have not been a phish anymore when the bot has visited them. The states table contains explanations to the states used in the websites table. The most important state is "3" all websites with this state are real phishing websites that have been assigned a parent website.

=== TABLE: Screenshots ===
The screeshots table contains information about all the screenshot-image files made. All those files are in the subfolder "screenshots" that is part of the distribution of this database.
Fields:
id: A unique id for each screenshot
website: the website this screenshot belongs to
type: the type of the screenshot taken (1=FULL all content of the website; 2=CONTENT croppped to the visible portion of the website; 3=WINDOW showing the whole browser window as it was rendered)
name: the name of the file as it is stored in the screenshot folder
width: image width in pixels
height: image height in pixels
md5: an md5 hash of the file contents. This may allow to compare files that have exactly the same content

=== TABLE: Websites ===                                                                                  
The websites table contains all the information that was collected for the websites. Not all the fields may be applicable for all websites
id: a unique id for the website
alexarank: for websites paresed from the top 1000 of alexa.com this is the rank of the websites, otherwise null
isPhish: is this webpage url from a phishinglist (1) or non-fraudulent (0)
parent: what is the parent website for this website (for phishes this contains the verified original website) otherwise null
parentCount: A counter how many parents have been found for this website (this is redundant as it could be calculated counting all websites that have their parent field set to the id of the website)
url: the url that was originally provided for the scan
urlHash: an md5 hash of this url for quicker finding of identical urls
urlBasedomain: the basedomain of this url (this usually means the top-level domain plus the domain part in front of it e.g. "google.com" for some special domain names this may include some more e.g. "google.co.uk")
finalUrl: the url that was finally parsed after following all redirect requests
finalUrlBasedomain: same as urlBaseDomain for finalUrl
name: a name identifiert that is sometimes assigned to websites (otherwise null)
quality: reserverd for future use (always null)
scanned: A UNIX-Timestamp when this url was visited
rescan: used internally for rescanning already scanned website (always 0)
statusCode: the statusCode that was returned for this website. Only websites with 200 were manually checked and assigned to states
htmlContent: the HTML content of the page that was loaded under finalUrl
loadTime: The time it took querying the content and taking all screenshots
phishtank_XYZ: fields parsed from the phishtank.com list of fraudulent websites. 
state: The state of the phishing website (3 ist most important here (see states table for more information))
duplicatedFrom: in some cases websites have been assigned more quickly by assigning them as duplicated to another phishing websites. This was stored here. This field does NOT denote that those websites are all duplicated that can be found for that website neither does it denote that the websites look 100percent simmilar.
created: UNIX-Timestamp when this website was created and stored in the table (usually prior to scanned)  