17-01-2013, 02:42 PM
Ranking Model Adaptation for Domain-Specific Search
1Ranking Model.pdf (Size: 1.36 MB / Downloads: 41)
Abstract
With the explosive emergence of vertical search domains, applying the broad-based ranking model directly to different
domains is no longer desirable due to domain differences, while building a unique ranking model for each domain is both laborious for
labeling data and time consuming for training models. In this paper, we address these difficulties by proposing a regularization-based
algorithm called ranking adaptation SVM (RA-SVM), through which we can adapt an existing ranking model to a new domain, so that
the amount of labeled data and the training cost is reduced while the performance is still guaranteed. Our algorithm only requires the
prediction from the existing ranking models, rather than their internal representations or the data from auxiliary domains. In addition, we
assume that documents similar in the domain-specific feature space should have consistent rankings, and add some constraints to
control the margin and slack variables of RA-SVM adaptively. Finally, ranking adaptability measurement is proposed to quantitatively
estimate if an existing ranking model can be adapted to a new domain. Experiments performed over Letor and two large scale data
sets crawled from a commercial search engine demonstrate the applicabilities of the proposed ranking adaptation algorithms and the
ranking adaptability measurement.
INTRODUCTION
LEARNING to rank is a kind of learning-based information
retrieval techniques, specialized in learning a ranking
model with some documents labeled with their relevancies
to some queries, where the model is hopefully capable of
ranking the documents returned to an arbitrary new query
automatically. Based on various machine learning methods,
e.g., Ranking SVM [12], [14], RankBoost [9], RankNet [4],
ListNet [5], LambdaRank [3], etc., the learning to rank
algorithms have already shown their promising performances
in information retrieval, especially web search.
However, as the emergence of domain-specific search
engines, more attentions have moved from the broad-based
search to specific verticals, for hunting information constraint
to a certain domain. Different vertical search engines
deal with different topicalities, document types or domainspecific
features. For example, a medical search engine
should clearly be specialized in terms of its topical focus,
whereas a music, image or video search engine would
concern only the documents in particular formats.
RANKING ADAPTATION
We define the ranking adaptation problem formally as
follows: for the target domain, a query set Q ¼ fq1; q2;
. . . ; qMg and a document set D ¼ fd1; d2; . . . ; dNg are given.
For each query qi 2 Q, a list of documents di ¼ fdi1; di2; . . . ;
di;nðqiÞg are returned and labeled with the relevance degrees
yi ¼ fyi1; yi2; . . . ; yi;nðqiÞg by human annotators. The relevance
degree is usually a real value, i.e., yij 2 IR, so that
different returned documents can be compared for sorting
an ordered list. For each query document pair <qi; dij>, an sdimensional
query dependent feature vector ðqi; dijÞ 2 IRs
is extracted, e.g., the term frequency of the query keyword qi
in the title, body, URL of the document dij. Some other
hyperlink based static rank information is also considered,
such as Pagerank [21], HITS [17], and so on. nðqiÞ denotes the
number of returned documents for query qi. The target of
learning to rank is to estimate a ranking function f 2 IRs !
IR so that the documents d can be ranked for a given query q
according to the value of the prediction fððq; dÞÞ.
Ranking Adaptation SVM
It can be assumed that, if the auxiliary domain and the
target domain are related, their respective ranking functions
fa and f should have similar shapes in the function space
IRs ! IR. Under such an assumption, fa actually provides a
prior knowledge for the distribution of f in its parameter
space. The conventional regularization framework, such as
Lp-norm regularization, manifold regularization [1] designed
for SVM [27], regularized neural network [11], and
so on, shows that the solution of an ill-posed problem can
be approximated from variational principle, which contains
both the data and the prior assumption [11]. Consequently,
we can adapt the regularization framework which utilizes
the fa as the prior information, so that the ill-posed problem
in the target domain, where only few query document pairs
are labeled, can be solved elegantly.
EXPLORE RANKING ADAPTABILITY
Though the ranking adaptation can mostly provide benefits
for learning a new model, it can be argued that when the
data from auxiliary and target domains share little common
knowledge, the auxiliary ranking model can provide little
help or even negative influence, to the ranking of the
documents in the target domain. Consequently, it is
imperative to develop a measure for quantitatively estimating
the adaptability of the auxiliary model to the target
domain. However, given a ranking model and a data set
collected for a particular target domain, it’s nontrivial to
measure their correlations directly, because neither the
distribution of the ranking model nor that of the labeled
samples in the target domain is trivial to be estimated. Thus,
we present some analysis on the properties of the auxiliary
model, based on which the definition of the proposed
ranking adaptability is presented.
RANKING ADAPTATION WITH DOMAIN-SPECIFIC
FEATURE
Conventionally, data from different domains are also
characterized by some domain-specific features, e.g., when
we adopt the ranking model learned from the webpage
search domain to the image search domain, the image content
can provide additional information to facilitate the text-based
ranking model adaptation. In this section, we discuss how to
utilize these domain-specific features, which are usually
difficult to translate to textual representations directly, to
further boost the performance of the proposed RA-SVM.
The basic idea of our method is to assume that
documents with similar domain-specific features should
be assigned with similar ranking predictions. We name the
above assumption as the consistency assumption, which
implies that a robust textual ranking function should
perform relevance prediction that is consistent to the
domain-specific features.
To implement the consistency assumption, we are
inspired by the work [26] and recall that for RA-SVM in
(2), the ranking loss is directly correlated with the slack
variable, which stands for the ranking loss for pairwise
documents, and is nonzero as long as the ranking function
predicts a wrong order for the two documents. In addition,
as a large margin machine, the ranking loss of RA-SVM is
also correlated with the large margin specified to the
learned ranker. Therefore, to incorporate the consistency
constraint, we rescale the ranking loss based on two
strategies, namely margin rescaling and slack rescaling.
The rescaling degree is controlled by the similarity between
the documents in the domain-specific feature space, so that
similar documents bring about less ranking loss if they are
ranked in a wrong order. We discuss the detailed formulations
of margin rescaling and slack rescaling as follows.
EXPERIMENTS
In this section, we perform several experiments under two
different settings, to demonstrate the effectiveness of the
proposed RA-SVM-based algorithms and the ranking adaptability
measurement.
Data Sets and Evaluation Measure
We first conduct the experiments over the Letor benchmark
data set [20], and adapt the ranking model learned
from TD2003 data set to the ranking of TD2004 data set.
Letor TD2003 and TD2004 data sets are gathered from the
topic distillation task of TREC 2003 and TREC 2004, with
50 queries for TD2003 and 75 ones for TD2004. The
documents are collected by crawling from the .gov
domain. For each query, about 1,000 associated documents
are returned, and labeled with a binary judgment, i.e.,
relevant or irrelevant. The features of TD2003 and TD2004
include the low-level features such as term frequency,
inverse document frequency, and document length, as well
as high-level features such as BM25, LMIR, PageRank, and
HITS, for totally 44 dimensional features. However, Letor
is a comparatively small data set, and each document is
only labeled with a binary relevance degree, which cannot
reflect the practical web search scenarios with multiple
relevance degrees. Also, there are no domain-specific
features for the target domain data, where we cannot
demonstrate the effectiveness of the proposed ranking
adaptation with domain-specific feature algorithms.
RELATED WORK
We present some works that closely relate to the concept of
ranking model adaptation here. To create a ranking model
that can rank the documents according to their relevance to
a given query, various types of models have been proposed,
some of which have even been successfully applied to web
search engines. Classical BM25 [24] and Language Models
for Information Retrieval (LMIR) [19], [23] work quite stable
for the broad-based search with few parameters needing
adjusted. However, with the development of statistical
learning methods, and more labeled data with complicated
features being available, sophisticated ranking models
become more desirable for achieving better ranking
performance. Recently, a dozen of learning to rank
algorithms based on machine learning techniques have
been proposed. Some of them transform the ranking
problem into a pairwise classification problem, which takes
a pair of documents as a sample, with the binary label taken
as the sign of the relevance difference between the two
documents, e.g., Ranking SVM [12], [14], RankBoost [9],
RankNet [4], and, etc. Some other methods including
ListNet [5], SVMMap [31], AdaRank [28], PermuRank [29],
LambdaRank [3], and, etc., focus on the structure of ranking
list and the direct optimization of the objective evaluation
measures such as Mean Average Precision and Normalized
Discounted Cumulative Gain.
CONCLUSION
As various vertical search engines emerge and the amount
of verticals increases dramatically, a global ranking model,
which is trained over a data set sourced from multiple
domains, cannot give a sound performance for each specific
domain with special topicalities, document formats, and
domain-specific features. Building one model for each
vertical domain is both laborious for labeling the data and
time consuming for learning the model. In this paper, we
propose the ranking model adaptation, to adapt the welllearned
models from the broad-based search or any other
auxiliary domains to a new target domain. By model
adaptation, only a small number of samples need to be
labeled, and the computational cost for the training process
is greatly reduced.