Schema-Based Re-Evaluation Of Knowledge Graph Embeddings

Description

Knowledge graph embedding(KGE) models play an important role in many downstream tasks such as question answering, dialog system, and social network clustering. Most of existing knowledge graph embedding models adopt link prediction tasks as evaluation methods. Given incomplete triples such as (h,r,?) and (?,r,t), link prediction task will predict the missing entities based on learned entity and relation representations. In this process, the relational structure of entities are evaluated. However, for many downstream tasks, such as recommendation, social network clustering, not only the relational information in knowledge graph but also the semantic similarities between entities are important. Despite their importance, corresponding evaluation methods are neglected for most knowledge graph embeddings. Nevertheless, another type of knowledge graph's structure information neglected by current evaluation tasks, the schema, defines high-level terms used in the knowledge graph and provides hierarchical and categorical information of entities. Therefore, we propose a schema-based evaluation task for additional semantic evaluation of knowledge graph embedding models. Based on the schema information of Freebase, we create a new dataset suitable for entity classification evaluation task and conduct an extensive re-evaluation on various knowledge graph embedding methods. The obtained results provide a new insight on existing knowledge graph embedding models.