User modeling for evaluation of reputation systems

To evaluate the proposed belief propagation based iterative trust and repuation management (BP-ITRM) algorithm, we generate ratings for service providers in the reputation systems through user modeling method to mimic the behavior of both non-malicious and malicious users. The detailed user model description is provided in our recent publication on reputation systems [1]. Our next step is to further evaluate BP-ITRM using real-life datasets such as those collected from eBay, Advogato, and Epinions.

  1. E. Ayday and F. Fekri, "Iterative trust and reputation management using belief propagation," IEEE Transactions on Dependable and Secure Computing, vol. 9, no. 3, pp. 375-386, 2012. [pdf]

 

Real-life data sets for evaluation of recommender systems

There are a couple of popular real-life rating data sets such as Netflix and MovieLens that are publicly available for evaluation of recommedation algorithms. We used the MovieLens data, collected from the MovieLens web site by GroupLens research lab at the University of Minnesota.

 

Evaluation by user study

We will set up a user study for evaluation and implementation of reputation and recommender systems since validation and improvement of our methods is mainly based on the user experiences of the prototype system. To achieve this goal, we are planning to develop a web-based virtual education portal InfoXchange@Gatech at the Georgia Tech campus, which will be an online information exchange platform.

 

Shilling attack dataset

We evaluate the performance of the proposed BP-based attack detection algorithm using the 100K MovieLens dataset. The dataset contains 100,000 ratings, all integers from 1 to 5, on 1682 items (movies) by 943 users. We treat the original users in the dataset as genuine users. To launch shilling attacks, a number of spam users are injected into the system. In particular, the Average attack model is adopted, where the rating on each filler item follows a normal distribution with its mean set as the average rating received by the filler item. Finally, a set of items are selected as targets in the attack.

 

Top-N recommendation in social networks

We evaluate the top-N recommendation performance of the proposed EP algorithm using the Epinions dataset. The dataset consists of 49,290 users and 139,738 items. A total of 664,824 ratings are given by users on items, and each rating is an integer between 1 and 5. The dataset also includes 487,181 directed trust statements with trust value one.

 

Trust inference in social networks

We use the Twitter dataset consisting of the real Twitter user profiles. The spam users are identified as those post URL links of malicious websites, e.g., phishing websites, by the SUCCESS research lab at the Texas A&M University. The crawled data for each user include the basic user profile information, e.g., account creation time, the list of followers and followings, and the most recent Tweets.

 

Link recommendation in Twitter networks

We evaluate the empirical performance on the Twitter dataset and the Tencent Weibo dataset. Tencent Weibo is a Twitter-like online microblogging service widely used in China. The Twitter dataset is downloaded from Twitter.com using Twitter API. The Tencent Weibo dataset is a subset of the dataset provided by KDD Cup 2012 Track 1.