|
<!DOCTYPE html> |
|
<html lang="en"> |
|
<head> |
|
<meta charset="UTF-8"> |
|
<meta name="viewport" content="width=device-width, initial-scale=1.0"> |
|
<title>Stick To Your Role! Leaderboard</title> |
|
|
|
<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/5.1.3/css/bootstrap.min.css"> |
|
|
|
<link rel="stylesheet" href="https://cdn.datatables.net/1.11.5/css/dataTables.bootstrap5.min.css"> |
|
|
|
<style> |
|
html, body { |
|
height: 100%; |
|
overflow: auto; |
|
} |
|
|
|
body { |
|
background-color: #f8f9fa; |
|
font-family: 'Arial', sans-serif; |
|
} |
|
.container { |
|
max-width: 1200px; |
|
margin: auto; |
|
padding: 15px; |
|
background: #fff; |
|
border-radius: 8px; |
|
box-shadow: 0 4px 8px rgba(0,0,0,0.1); |
|
} |
|
|
|
.container h1 { |
|
color: #333; |
|
text-align: center; |
|
} |
|
|
|
.container h3 { |
|
margin-top: 10px; |
|
margin-bottom: 50px; |
|
margin-left: 20px; |
|
margin-right: 20px; |
|
text-align: center; |
|
} |
|
|
|
p { |
|
margin: auto; |
|
margin-top: 20px; |
|
margin-bottom: 10px; |
|
max-width: 1000px; |
|
text-align: left; |
|
} |
|
|
|
ul { |
|
margin: auto; |
|
margin-top: 20px; |
|
margin-bottom: 10px; |
|
max-width: 1000px; |
|
text-align: left; |
|
list-style-type: disc; |
|
padding-left: 20px; |
|
} |
|
|
|
ul li { |
|
margin-bottom: 10px; |
|
} |
|
|
|
|
|
.table-responsive { |
|
margin-top: 20px; |
|
max-width: 1000px; |
|
margin: auto; |
|
} |
|
|
|
.main-table { |
|
font-size: 15px |
|
} |
|
|
|
.full-table { |
|
font-size: 12px |
|
} |
|
|
|
table { |
|
border-collapse: separate; |
|
border-spacing: 0; |
|
width: 1000px; |
|
margin: auto; |
|
border: none; |
|
} |
|
table thead th { |
|
background-color: #610b5d; |
|
color: white; |
|
border: 1px solid #dee2e6; |
|
text-align: left; |
|
} |
|
table tbody tr { |
|
background-color: #fff; |
|
box-shadow: 0 2px 4px rgba(0,0,0,0.1); |
|
} |
|
table tbody tr:hover { |
|
background-color: #f1f1f1; |
|
} |
|
table td, table th { |
|
padding: 5px; |
|
border: 1px solid #dee2e6; |
|
} |
|
table th:first-child { |
|
border-top-left-radius: 10px; |
|
} |
|
table th:last-child { |
|
border-top-right-radius: 10px; |
|
} |
|
.section { |
|
padding-left: 150px; |
|
padding-right: 150px; |
|
text-align: left; |
|
} |
|
.citation-section { |
|
margin-top: 5px; |
|
text-align: center; |
|
max-width: 1000px; |
|
margin: auto; |
|
} |
|
.citation-box { |
|
background-color: #f8f9fa; |
|
border: 1px solid #dee2e6; |
|
border-radius: 8px; |
|
padding: 5px; |
|
margin-top: 5px; |
|
font-size: 13px; |
|
text-align: left; |
|
font-family: 'Courier New', Courier, monospace; |
|
white-space: pre; |
|
} |
|
.image-container { |
|
display: flex; |
|
justify-content: center; |
|
gap: 10px; |
|
margin-bottom: 40px; |
|
max-width: 1100px; |
|
margin: auto; |
|
} |
|
.image-container a { |
|
flex: 1; |
|
} |
|
.image-container img { |
|
max-width: 100%; |
|
height: auto; |
|
display: block; |
|
margin: auto; |
|
} |
|
.about-button { |
|
text-align: center; |
|
margin-top: 50px; |
|
margin-bottom: 50px; |
|
} |
|
.custom-button { |
|
background-color: #610b5d; |
|
color: #fff; |
|
border-radius: 15px; |
|
padding: 10px 20px; |
|
font-size: 18px; |
|
text-decoration: none; |
|
} |
|
.custom-button:hover { |
|
background-color: #812b7d; |
|
color: #fff; |
|
} |
|
|
|
|
|
table.dataTable thead .sorting:after, |
|
table.dataTable thead .sorting:before, |
|
table.dataTable thead .sorting_asc:before, |
|
table.dataTable thead .sorting_asc:after, |
|
table.dataTable thead .sorting_desc:before, |
|
table.dataTable thead .sorting_desc:after { |
|
display: none; |
|
} |
|
|
|
table.dataTable thead .sorting_asc { |
|
background-image: url("{{ url_for('static', filename='icons/sort_asc_gray.png') }}"); |
|
background-repeat: no-repeat; |
|
background-position: center right; |
|
} |
|
|
|
table.dataTable thead .sorting_desc { |
|
background-image: url("{{ url_for('static', filename='icons/sort_desc_gray.png') }}"); |
|
background-repeat: no-repeat; |
|
background-position: center right; |
|
} |
|
|
|
|
|
table.dataTable > thead > tr > th:not(.sorting_disabled), |
|
table.dataTable > thead > tr > td:not(.sorting_disabled) { |
|
padding-right: 5px; |
|
} |
|
|
|
|
|
table.dataTable tbody td:first-child, |
|
table.dataTable tbody td:nth-child(2), |
|
table.dataTable thead th:first-child, |
|
table.dataTable thead th:nth-child(2) { |
|
text-align: left; |
|
} |
|
|
|
|
|
table.dataTable tbody td:not(:first-child):not(:nth-child(2)), |
|
table.dataTable thead th:not(:first-child):not(:nth-child(2)) { |
|
text-align: center; |
|
} |
|
|
|
|
|
.tooltip-inner { |
|
max-width: none; |
|
} |
|
|
|
th[title] { |
|
position: relative; |
|
cursor: help; |
|
} |
|
|
|
th[title]:hover::after { |
|
content: attr(title); |
|
position: absolute; |
|
bottom: 100%; |
|
left: 50%; |
|
transform: translateX(-50%); |
|
background-color: #333; |
|
color: white; |
|
padding: 5px 10px; |
|
border-radius: 4px; |
|
white-space: nowrap; |
|
z-index: 1; |
|
font-weight: normal; |
|
font-size: 14px; |
|
} |
|
</style> |
|
</head> |
|
<body> |
|
<div class="container"> |
|
<h1 class="mt-5">Stick To Your Role! Leaderboard</h1> |
|
<h3> |
|
LLMs can role-play different personas by simulating their values and behavior, but can they stick to their role whatever the context? |
|
Is simulated Joan of Arc more tradition-driven than Elvis? |
|
Will it still be the case after playing chess? |
|
</h3> |
|
<p> |
|
The Stick to Your Role! leaderboard compares LLMs based on <b>undesired sensitivity to context change</b>. |
|
LLM-exhibited behavior always depends on the context (prompt). |
|
While some context-dependence is desired (e.g. following instructions), |
|
some is undesired (e.g. drastically changing the simulated value expression based on the interlocutor). |
|
As proposed in our <a target="_blank" href="https://arxiv.org/abs/2402.14846">paper</a>, |
|
undesired context-dependence should be seen as a <b>property of LLMs</b> - a dimension of LLM comparison (alongside others such as model size speed or expressed knowledge). |
|
This leaderboard aims to provide such a comparison and extends our paper with a more focused and elaborate experimental setup. |
|
Standard benchmarks present <b>many</b> questions from the <b>same minimal contexts</b> (e.g. multiple choice questions), |
|
we present <b>same</b> questions from <b>many different contexts</b>. |
|
</p> |
|
<p> |
|
The Stick to You Role! leaderboard focuses on the <b>stability of simulated personal values during role-playing</b>. |
|
We study the <b>coherence of a simulated population</b>. |
|
In contrast to evaluating each simulated persona separately, we evaluate personas relative to each other, i.e. as a population. |
|
You can browse the simulated population, questionnaires, and contexts used on our <a target="_blank" href="https://huggingface.co/datasets/flowers-team/StickToYourRole">🤗 StickToYourRole dataset</a>. |
|
</p> |
|
<div class="table-responsive main-table"> |
|
|
|
{{ main_table_html|safe }} |
|
</div> |
|
<div class="image-container"> |
|
<a href="{{ url_for('static', filename='models_data/ordinal.svg') }}" target="_blank"> |
|
<img src="{{ url_for('static', filename='models_data/ordinal.svg') }}" alt="Ordinal"> |
|
</a> |
|
<a href="{{ url_for('static', filename='models_data/cardinal.svg') }}" target="_blank"> |
|
<img src="{{ url_for('static', filename='models_data/cardinal.svg') }}" alt="Cardinal"> |
|
</a> |
|
</div> |
|
<p> |
|
We leverage Schwartz's theory of <a href="https://www.sciencedirect.com/science/article/abs/pii/S0065260108602816">Basic Personal Values</a>, |
|
which defines 10 values Self-Direction, Stimulation, Hedonism, Achievement, Power, Security, Conformity, Tradition, Benevolence, Universalism), |
|
and the associated PVQ-40 and SVS questionnaires (available <a href="https://www.researchgate.net/publication/354384463_A_Repository_of_Schwartz_Value_Scales_with_Instructions_and_an_Introduction">here</a>). |
|
</p> |
|
<p> |
|
Using the <a href="https://pubmed.ncbi.nlm.nih.gov/31402448/">methodology from psychology</a>, we focus on population-level (interpersonal) value stability, i.e. <b>Rank-Order stability (RO stability)</b>. |
|
Rank-Order stability refers to the extent to which the order of different personas (in terms of expression of some value) remains the same along different contexts. |
|
Refer <a href="{{ url_for('about', _anchor='rank_order_stability') }}">here</a> or to our <a href="https://arxiv.org/abs/2402.14846">paper</a> for more details. |
|
</p> |
|
<p> |
|
In addition to Rank-Order stability we compute <b>validity metrics (Stress, CFI, SRMR, RMSEA)</b>, which are a common practice in psychology. |
|
Validity refers to the extent to which the questionnaire measures what it purports to measure. |
|
It can be seen as the questionnaire's accuracy in measuring the intended factors, i.e. values. |
|
For example, basic personal values should be organized in a circular structure, and questions measuring the same value should be correlated. |
|
The table below additionally shows the validity metrics, refer <a href="{{ url_for('about', _anchor='metrics') }}">here</a> for more details. |
|
</p> |
|
<p> |
|
We <b>aggregate</b> Rank-Order stability and validation metrics to rank the models. We do so in two ways: <b>Cardinal</b> and <b>Ordinal</b>. |
|
Following <a href="https://arxiv.org/abs/2405.01719">this paper</a>, we compute the stability and diversity of those rankings. See <a href="{{ url_for('about', _anchor='aggregate_metrics') }}">here</a> for more details. |
|
</p> |
|
<p> |
|
To sum up here are the metrics used: |
|
<ul> |
|
<li><b>RO-stability</b> - |
|
<i>Do the same simulated participants always (in every context) express same values more strongly than other participants?</i> |
|
|
|
|
|
<div style="margin-left: 20px; margin-top: 5px"> |
|
The correlation in the order of simulated participants (ordered based on the expression of the same values) over different contexts |
|
</div> |
|
</li> |
|
|
|
<li><b>Stress</b> - |
|
<i>Is value expression (intercorrelations of values) structured as predicted by the theory (and as in humans), i.e. in a circle?</i> |
|
<div style="margin-left: 20px; margin-top: 5px"> |
|
The Multi-dimensional scaling (MDS) fit of the observed value structure to the theoretical circular structure. Stress of 0 indicates 'perfect' fit, 0.025 excellent, 0.05 good, 0.1 fair, and 0.2 poor. |
|
</div> |
|
</li> |
|
|
|
<li><b>CFI, SRMR, RMSEA</b> - |
|
<i>To what extent does the questionnaire measure what it is supposed to measure - values?</i> |
|
<div style="margin-left: 20px; margin-top: 5px"> |
|
Common Confirmatory Factor Analysis (CFA) metrics showing the fit of the posited model of the relation of items (questions) to factors (values) on the observed data, applied here with Magnifying Glass CFA. For CFI >.90 is considered acceptable fit, for SRMR and RMSEA is <.05 considered good fit and <.08 reasonable. |
|
</div> |
|
</li> |
|
|
|
<li><b>Ordinal - Win Rate</b> - |
|
<i>Which model beats the most other models across most metrics?</i> |
|
<div style="margin-left: 20px; margin-top: 5px"> |
|
The score averaged over all metrics (with descending metrics inverted), context pairs (for stability) and contexts (for validity metrics) |
|
<div> |
|
</li> |
|
<li><b>Cardinal - Score</b> - |
|
<i>Which model has the highest average score?</i> |
|
<div style="margin-left: 20px; margin-top: 5px"> |
|
The percentage of won games, where a game is a comparison of each model pair, each metric, and each context pair (for stability) or context (for validity metrics) |
|
</div> |
|
</li> |
|
</ul> |
|
</p> |
|
<div class="table-responsive full-table"> |
|
|
|
{{ full_table_html|safe }} |
|
</div> |
|
<div class="about-button"> |
|
<a href="{{ url_for('about') }}" class="custom-button mt-3">Motivation and Methodology page</a> |
|
<a href="{{ url_for('new_model') }}" class="custom-button mt-3">Submit a model</a> |
|
</div> |
|
<div class="citation-section"> |
|
<p> |
|
You can find more details in our <a target="_blank" href="https://arxiv.org/abs/2402.14846">paper</a>. |
|
</p> |
|
<p> |
|
If you found this project useful, please cite one of our related papers, |
|
which this leaderboard extends with a more focused and elaborate experimental setup. |
|
Refer to the <a href="{{ url_for('about', _anchor='paper') }}">site</a> for details. |
|
</p> |
|
<p>Short paper: <a target="_blank" href="https://escholarship.org/uc/item/7w4823c6">Kovač, G., Portelas, R., Sawayama, M., Dominey, P. F., & Oudeyer, P. Y. (2024). Stick to your Role! Stability of Personal Values Expressed in Large Language Models. In Proceedings of the Annual Meeting of the Cognitive Science Society (Vol. 46).</a> </p> |
|
<div class="citation-box" id="citation-text"> |
|
@inproceedings{kovavc2024stick, |
|
title={Stick to your Role! Stability of Personal Values Expressed in Large Language Models}, |
|
author={Kova{\v{c}}, Grgur and Portelas, R{\'e}my and Sawayama, Masataka and Dominey, Peter Ford and Oudeyer, Pierre-Yves}, |
|
booktitle={Proceedings of the Annual Meeting of the Cognitive Science Society}, |
|
volume={46}, |
|
year={2024} |
|
} |
|
</div> |
|
<p>Longer paper: <a target="_blank" href="https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0309114"> Kovač G, Portelas R, Sawayama M, Dominey PF, Oudeyer PY (2024) Stick to your role! Stability of personal values expressed in large language models. PLOS ONE 19(8): e0309114. https://doi.org/10.1371/journal.pone.0309114 </a></p> |
|
<div class="citation-box" id="citation-text"> |
|
@article{kovavc2024stick, |
|
title={Stick to your role! Stability of personal values expressed in large language models}, |
|
author={Kova{\v{c}}, Grgur and Portelas, R{\'e}my and Sawayama, Masataka and Dominey, Peter Ford and Oudeyer, Pierre-Yves}, |
|
journal={PloS one}, |
|
volume={19}, |
|
number={8}, |
|
pages={e0309114}, |
|
year={2024}, |
|
publisher={Public Library of Science San Francisco, CA USA} |
|
} |
|
</div> |
|
</div> |
|
<ul> |
|
<li>Contact: <a href="mailto: [email protected]">[email protected]</a></li> |
|
<li>See the <a target="_blank" href="https://sites.google.com/view/llmvaluestability">Project website</a></li> |
|
<li>See the Flowers team <a target="_blank" href="http://developmentalsystems.org">blog</a> and <a target="_blank" href="https://flowers.inria.fr/">website</a></li> |
|
<li>See Grgur's website and other projects: <a target="_blank" href="https://grgkovac.github.io">https://grgkovac.github.io</a></li> |
|
</ul> |
|
</div> |
|
|
|
|
|
<script src="https://code.jquery.com/jquery-3.6.0.min.js"></script> |
|
|
|
<script src="https://stackpath.bootstrapcdn.com/bootstrap/5.1.3/js/bootstrap.bundle.min.js"></script> |
|
|
|
<script src="https://cdn.datatables.net/1.11.5/js/jquery.dataTables.min.js"></script> |
|
<script src="https://cdn.datatables.net/1.11.5/js/dataTables.bootstrap5.min.js"></script> |
|
|
|
<script> |
|
$(document).ready(function() { |
|
const table = $('table').DataTable({ |
|
"paging": false, |
|
"info": false, |
|
"columnDefs": [ |
|
{ "orderable": false, "targets": 0 }, |
|
{ "searchable": false, "targets": 0 } |
|
], |
|
"order": [[ 2, 'desc' ]], |
|
"drawCallback": function(settings) { |
|
var api = this.api(); |
|
api.column(0, {order:'applied'}).nodes().each(function(cell, i) { |
|
cell.innerHTML = i + 1; |
|
}); |
|
} |
|
}); |
|
}); |
|
</script> |
|
</body> |
|
</html> |
|
|
