diff --git a/test/benchmarks/README.md b/test/benchmarks/README.md
new file mode 100644
index 000000000..ba06be2ad
--- /dev/null
+++ b/test/benchmarks/README.md
@@ -0,0 +1,20 @@
+# Benchmarks
+
+Run the benchmarks with the following command:
+
+```
+python run.py [--reader] [--retriever_index] [--retriever_query] [--ci] [--update-json]
+```
+
+You can specify which components and processes to benchmark with the following flags.
+
+**--reader** will trigger the speed and accuracy benchmarks for the reader. Here we simply use the SQuAD dev set.
+
+**--retriever_index** will trigger indexing benchmarks
+
+**--retriever_query** will trigger querying benchmarks (embeddings will be loaded from file instead of being computed on the fly)
+
+**--ci** will cause the the benchmarks to run on a smaller slice of each dataset and a smaller subset of Retriever / Reader / DocStores. 
+
+**--update-json** will cause the script to update the json files in docs/_src/benchmarks so that the website benchmarks will be updated.
+ 
\ No newline at end of file