Context: The interest in developing chatbots is on the rise as the usability evaluation is an essential step in the chatbot development process; the number of experimental studies of chatbot usability has grown as well.Objective: Aggregating and concluding the features and metrics used to evaluate the usability of chatbots in experiments, to identify the state of the art of chatbots usability experimentation.Method: A systematic mapping study has been conducted, searching in five scientific databases.Results: Of 363 papers, 14 papers with experiments were selected as the primary studies.The published works in this area were initiated in 2018.Control tools are applied commonly in experiments.Various advantages and shortages of chatbot usability experiments were revealed, for example, most of the experiments do not provide raw data, and only one of the identified works replicated the experiment.Conclusions: An increased interest in usability experimentation of chatbots is observed in recent years.The chatbot usability experiment should be more replicable to improve the reliability of experimental results.